Getting data in

Getting data into your Brinqa application requires configuring several elements via the administration panel. Data server configuration establishes a secure connection between on-premise data sources and the Brinqa cloud. Data sources allow administrators to configure the services that will provide data via those servers. Data mappings determine how source attributes are matched to Brinqa attributes in data models, which serve as the schema for incoming data and determine how objects like vulnerabilities are created. Scheduled syncs and rules then determine when data is imported and exported from the system.

This article will walk through the configuration process for getting data into your application.

Determine what data you want to bring into the system¶

Before configuring any settings to bring data in, it's best to determine exactly what data you'll want to work with in your Brinqa application. Objects in the system (i.e. datasets, like particular hosts or vulnerabilities) are created when a data source provides data to a data model, which structures the data into a Brinqa object. Knowing what objects you want and what attributes they should have will clarify which data sources should be configured and how they should be mapped.

Start by reviewing the default data models that come with the application and determining whether the objects they create will be sufficient for your needs. Typical objects include vulnerabiltiies, hosts, tickets, applications, and issues. A list of default data models and their attributes can be found in the Default data models article. Default data models can be found in the application by navigating to Administration > Data management > Data models.

You may want to add more attributes to an existing data model or create a new data model entirely to ensure your system is populated with the right objects or attributes for those objects. Steps for doing that can be found in the Attribute and Data Model articles linked above.

You can also review the table below and identify which data sources you have and what data models they typically populate data to.

Table 1. Data sources and data models

Install connector agents and configure data servers¶

Configuring data servers is the first step in integrating data, because they tell the system where to retrieve data from and are referenced when creating data sources.

Your Brinqa application will come with one pre-configured data server: "local server". When configuring data sources, "local server" is the data server you select for cloud data sources. Cloud data sources can connect directly to the Brinqa cloud to transmit data, whereas on-premise data sources require the establishment of a secure connection to the cloud.

To get data from on-premise servers to the Brinqa cloud, a connector agent must be installed on an on-premise server. This server must be on the same network segment as the server hosting the data to be imported. Data servers are created in Brinqa applications in order to generate a config file that can be placed on the on-premise connector agent server. The presence of the config file on this server associates it to the Brinqa system.

Procedure

1. Install the Brinqa connector agent on an on-premise server by unzipping the connector agent file to any directory on the server. The connector agent file can be obtained from Brinqa.
2. Navigate to Administration > Data integration > Data servers in your Brinqa application
3. Click Create data server
4. Enter a Title for the data server. The reference name for the data server will be automatically populated, but must be unique.
5. Enter a Description for the data server, e.g. a list of the service or file data it will provide
6. (Optional) Modify the Interval and Max. Connections properties. The interval determines how frequently the data server will ping the Brinqa cloud and max. connections determines how many connections the data server maintains in a pool. The default settings should be adequate for most situations.
7. (Optional) Check the Skip SSL verification box. This setting should be used only when initially testing connections.
8. Click Create
9. Navigate to Administration > Data integration > Data servers, if not already there
10. Mouse over the entry for the data server you created and click the Actions button that appears to the far right
11. Select "Download conf"
12. Place this file in the /conf folder in the directory where you installed the Brinqa Connector Agent, replacing the existing "agent.conf" file
13. Start the on-premise server
14. Repeat these steps for each connector agent server you have on-premise

For more information on data servers, refer to the Data server article.

Configure data sources¶

Once data servers have been configured, data sources can be created in the Brinqa application. Data source configuration involves telling the system where data from a particular service or file will be retrieved from (i.e. the cloud or an on-premise server) and what it needs to know to retrieve it, like which connector it should use, the URL of a cloud instance, or credentials for accessing the data.

Procedure

1. Navigate to Administration > Data integration > Data sources
2. Click the icon that corresponds with the desired data source under the Available Connectors heading. The connector serves as the schema for retrieving and interpretting data from that source. Each source uses a different connector.
3. Enter a Title for the data source
4. (Optional) Enter a Description for the data source
5. Confirm that the Connector field lists the connector you selected
6. Select the Data server that will handle data from this source. "Local server" should be selected for cloud data sources.
7. Fill in the connector properties. Each connector will have different properties specific to the data source being used.
8. Click Test Connection at the bottom of the form. You will receive a message saying whether or not the test was successful. If it wasn't, check the credentials entered in the connector properties. Accounts used must have at least read access to the data source.
9. Using the Accessible from field, choose whether data from this source should be available to all your Brinqa applications or only the application you are currently administrating.
10. Click Create
11. Repeat these steps for each data source

For more information on data sources, refer to the Data sources article.

Configure data mappings¶

Data mappings are how the system knows where to put the data it receives from data sources. They associate attributes in the source to attributes on the Brinqa data model, which structures objects that are created with that data in the system. Data cannot be synced until mappings have been configured, since without them the system doesn't know what to do with the data it receives.

The same data source may have multiple mappings, depending on the data it provides. For example, Qualys Vulnerability Management provides both host and vulnerability data, so requires a mapping for both hosts and vulnerabilities. The Qualys host attributes are mapped to the Brinqa host data model attributes, allowing for creation of host objects with the data from Qualys. Likewise, mapping the Qualys vulnerability attributes to the Brinqa vulnerability data model attributes allows for creation of vulnerability objects with the data from Qualys.

Procedure

1. Navigate to Administration > Data integration > Mappings
2. Click Create data mapping
3. Enter a Title for the mapping
4. Enter an Order. The order determines when this mapping will sync relative to other mappings on the same data source. It's especially relevant when working with host and vulnerability mappings--hosts should be synced first, since if new hosts are found, they must be created in the Brinqa application before vulnerabilities can be associated to them.
5. (Optional) Modify the default Options and Coalesce settings. Setting descriptions can be found here.
6. Select a Data source
7. Select a Source. The source is the name of objects for this mapping in the original data source.
8. Select a Target. The target is the data model in Brinqa that data from this source object should be mapped to.
9. Click Automap. This will automatically map whatever attributes share names in the data source and Brinqa data model.
10. (Optional) Map additional attributes that were not picked up by the automap. To do this, click Add attribute mapping and select Source and Target attributes to associate. You can also select the direction this field should sync, whether values should come from the data source (incoming), be pushed out by the Brinqa application (outgoing), or both (bidirectional).
11. (Optional) Add Transform Scripts and Sync operation options.
12. Using the Accessible from field, choose whether data from this source should be available to all your Brinqa applications or only the application you are currently administrating.
13. Click Create

For more information on data mappings, refer to the Data mapping article.

(Optional) Perform the first data sync¶

Data from data sources is not continuously synced to Brinqa applications, but rather synced periodically either manually or automatically. The first sync you perform can be scheduled, or done manually so that data appears in the system immediately. To perform a manual sync, follow the steps below.

Procedure

1. Navigate to Administration > Data integration > Data sources
2. Mouse over the Last synced date for the data source you want to sync in the list view at the top of the page. A small circular sync icon will appear.
3. Click the circular sync icon
4. Select a Sync data from option. This will determine how far back data is gathered. Syncs from "the beginning of time" can take a while to run, so should be done with caution.
5. Click Sync now

It's better to run syncs non-concurrently, so allow one sync to finish before performing further manual syncs, if you have multiple data sources.

Schedule recurring inbound data syncs¶

As mentioned in the previous step, data is not synced continuously to the system, but rather periodically synced either manually or automatically. Sync schedules allow you to specify times and conditions for automatic syncs. After the initial sync of data to the system, most syncs should be performed automatically with schedules.

Syncs should be scheduled during non-peak hours and at non-concurrently.

Procedure

1. Navigate to Administration > Data integration > Schedules
2. Click Create scheduled sync
3. Enter a Title
4. Enter a Data source that will be synced
5. Select a user to Run as
6. (Optional) Modify the Options settings
7. Select an interval for the sync to Run, e.g. daily. To chain syncs together so that they run back to back as each finishes, follow the tutorial found here.
8. Select a specific day/time/period for the Time
9. Using the Accessible from field, choose whether data from this source should be available to all your Brinqa applications or only the application you are currently administrating.
10. Click Create

For more information on scheduled syncs, refer to the Scheduled syncs article.

Configure outbound data sync rules¶

While not part of getting data into the Brinqa application, creating outbound sync rules is an important part of mainitaining consistent data across your systems. Outbound sync rules determine when data will be pushed from Brinqa applications out to your data sources. This is particularly helpful with tickets, where a ticket closing rule may close a ticket automatically when all its associated vulnerabilities have been marked closed. An outbound sync rule would then update the corresponding ticket in JIRA or ServiceNow automatically.

Procedure

1. Navigate to Administration > Data integration > Rules
2. Click Create rule
3. Enter a Title
4. Enter a Description. The description should include the event and condition under which the rule runs.
5. Select the Data model whose objects will be synced out
6. Enter an Order. The order will determine when this rule runs relative to other rules with the same events and data model.
7. Enter Events and Conditions under which the rule will run. For outbound data rules, the event will often be "After Update", since updating an object in the Brinqa application should ususally update it elsewhere.
8. Select the Data source that will receive the outbound data
9. Select the External ID attribute. This attribute should be the field that serves as a unique identifier for individual objects of this type. For example, the unique ticket number.
10. Using the Accessible from field, choose whether data from this source should be available to all your Brinqa applications or only the application you are currently administrating.
11. Click Create

For more information on data integration rules, refer to the Rules article.