Documentation for DEX products
The CDC Data Exchange (DEX) Upload API is an open-source service created to support public health Data Senders in their effort to share critical public health data with internal CDC Programs. The open-source model allows users to tailor the tool to fit specific data needs. \
The DEX Upload API service is a highly scalable, reliable means of transiting files of nearly any type and size from public health partners to the CDC, even when sent over unreliable network connections. As a current user, your organization will continue to benefit from the DEX Upload API’s robust list of features.
Resumable Uploads: Implementation of the Tus service as a library in a Golang application, facilitating high volume file uploads over HTTP.
Authentication: OAuth provider approval enforcement to ensure security.
Metadata Verification: Enforcement of upload submission metadata standards.
File Delivery: Routing of uploaded files to configured target destinations.
File Observability: Upload lifecycle event tracking and endpoints for health check, information, and application version.
Retry Delivery: Tool that delivers files to target destinations that uploaded successfully but were unsuccessful in delivery.
User Interface: Facilitates uploads and observability within a user interface.
The DEX Upload API accepts uploads using any client that implements the TUS protocol. Users can view example scripts for clients that can help them build their upload client correctly:
The DEX Upload API GitHub Repository has all the information you need to get set up. Review the Getting Started section and access information about:
Installation of required tools
A Tus client and Go are required; you will also need Docker or Podman.
An easy step to get your instance running.
A step-by-step instructions and commands for building, starting, and running the DEX Upload API.
The DEX Upload API supports configuration with Mac/Linux or Windows. You can review how to configure your instance in the Configuration section of the GitHub Repository and learn about:
General API configuration documentation
General configuration information can be found here; the env-configs section of the repository has more detailed information.
General code to support service configurations.
Configuring Distributed File Locking with Redis
To scale this service horizontally, you’ll need to use a distributed file-locking mechanism to prevent upload corruption. You can read more about the limitations of Tus’s support for concurrent requests here.
Current DEX Upload API customers use the DEX Upload API Metadata Json Schema to determine the data they collect and transfer to the CDC. The model includes:
| data_stream_id (example: celr) | Identifies the CDC program the file should be contained in |
| data_stream_route (example: hl7v2) | Defines the path that data takes through DEX to arrive at the CDC – this can be the name of the internal folder destination for the data file |
| sender_id (example: APHL) | Identifies the sender, machine, or intermediary that’s sending data |
| data_producer_id (example: FL-SPHL) | Identifies the public health authority that supplies the data |
| Jurisdiction (example: FL) | Defines the city, county, state, tribe, or territory where the data originates |
| received_filename (example: filename.hl7) | Identifies the name of the file being sent |
| Version (example: 2.0) | Defines the version of the metadata being sent |
| supporting_metadata | Metadata specific to your program that you track |
Your organization can model your metadata on the DEX Upload API metadata model fields above. As the DEX Upload API evolves, users will be able to create their own custom metadata fields.
Configuring file storage allows users with custom instances to decide where transferred files will be delivered. This is especially important for users with more than one delivery target. Read more about how to set up your file storage.
The DEX Upload API currently supports local file system, Azure, and AWS as storage backends. You can only use one storage backend at a time. If Azure configurations are set, it will be the default storage backend. If the Azure configurations are not set and the AWS S3 configurations are set, S3 will be the storage backend. If neither Azure nor S3 configurations are set, local storage will be the storage backend.
DEX Upload API routing defines how and where data will arrive once it has been transferred. Set up begins with the creation of a YML file that defines delivery groups, and one or more delivery targets. You can read more about configuring routing and delivery targets to make sure your data lands where you want it to.
DEX Upload API publishes reports and events. While both are published, they offer different benefits and are stored in different locations.
DEX Upload API reports offer observability so that users can understand what the application is doing with your data. By default reports:
You can set the LOCAL_REPORTS_FOLDER environment variable to set the location where these files get written. If you want to publish these reports to a separate service, you can configure a messaging broker like Azure Service Bus or AWS SNS/SQS.
You can read more about how to configure your reports location.
DEX Upload API events let the API know that a file has been uploaded and is ready to be delivered to its destination location. By default, event messages about upload and delivery activity are written to the ./uploads/events directory in the local file system.
You can choose to publish event messages to a message broker like Azure Service Bus or AWS SNS/SQS. Then, you can set environment variables with details related to the new message broker topic for future use in your application or environment. For detailed information about this process, read more about configuring the event publication and subscription.
The DEX Upload API uses smoke test suites that leverage kotlin (/smoke/kotlin) and playwright (/smoke/playwright).
Unit testing helps you catch issues early in development and ensure that individual parts of the API function correctly. Read more about how to run basic unit tests.
Ensure that all parts of the API are working together and functioning as expected. Read more about DEX Upload API integration testing.
The Upload API has OAuth token verification middleware for the /files/ and /info endpoints. You can read more detailed information about this topic here.
Currently, DEX Upload API customers use the Secure Access Management System (SAMS) Access to authenticate into the DEX Upload API environments. If your organization is a CDC public health partner, there is a good chance that you already have a SAMS account associated with your email or work with someone who does.
If you are not a public health partner of the CDC, you will need to wait until custom authorization and authentication methods are incorporated into the product (expected: TBD).
As the product evolves, the DEX Upload API will support standard OAuth2 resource provider behaviors, offering flexibility and security based on configuration. Users will be able to configure their own OAuth2 authentication using Okta or another token-based service. While this feature is not available today, check back for product updates.
The DEX Upload API has endpoints that can provide information about the status of your file uploads, transfers, and deliveries.
/info
This endpoint serves as a mechanism for you to verify that your upload has completed successfully and that they have been delivered to your chosen underlying storage location. You can send an HTTP GET request to this endpoint providing your unique ID of your upload as a path parameter.
/health
You can use the /health endpoint to check the status of the Upload API system, and its dependent systems, like your delivery target connection. Send an HTTP GET request to this endpoint and receive a 200 response with JSON in the body containing the overall status of the system, as well as statuses of dependent systems.
When an error occurs, all Tus clients provide feedback. Typically, this feedback arrives as exceptions or error handler callbacks, like a 404 error. The response will indicate the reason for the error. The following errors are the most common errors experienced by users.
Host forwarding mismatch errors arise when there is a discrepancy between the expected host or forwarding settings in your setup. This error usually indicates that the Tus server or the reverse proxy (like a load balancer) is misconfigured or that there’s an issue with the headers. To troubleshoot this error:
The following 400 responses indicate an issue with delivery:
409 Error
Two parallel PATCH requests could overwrite each other’s data. your Tus client gets a 409 response for parallel uploads, you need to upload the distributed locking. The DEX Upload API currently supports Redis. Read more about configuring Distributed File Locking with Redis.
If you need additional information or have questions, please contact the DEX Upload API Team.
| Report an incident: dexuploadapi@cdc.gov |
| Contact the team: dexuploadapi@cdc.gov |