Skip to main content

Set up Surface Discovery for GitHub

Surface Discovery for GitHub finds the APIs in every repository of a GitHub organization and registers each one as an APIsec application. One repository in your organization runs the APIsec Surface org scan workflow every night. The workflow runs in your organization, on your runners, with your secrets. With the settings in this guide, your source code stays on your runners. APIsec receives only the results of the Surface Code Discovery analysis: facts and observations about the code, such as routes, called functions, dependency versions, and masked secrets.

Your repository holds only a short caller workflow. The scan logic lives in a reusable workflow that APIsec hosts in the public repository apisec-inc/apisec-orchestrator. Fixes reach you on the next nightly run, with no change on your side.

Each run scans only the repositories that someone pushed to since their last successful scan. The first run scans every repository.

Unlike the single-repository workflow, Surface Discovery covers the whole organization from one repository.

Prerequisites​

  • An organization owner account on the GitHub organization. You need it to set organization-level Actions secrets and variables.
  • Your APIsec registration code, in the format ###-###, given to you by your APIsec Support resource. Each customer has their own code. If you don't have it, contact support.
  • An APIsec user account whose platform role is ADMIN or USER, to create the APIsec token.
  • A runner pool for the workflow. Use one of these:
    • GitHub-hosted runners. This is the default. No setup is needed.
    • Self-hosted runners registered to the organization with the custom label apisec. GitHub adds the self-hosted label automatically. The runners need Python 3.10 or later, git, and outbound access to GitHub, pypi.org, files.pythonhosted.org, and https://api.apisecapps.com.

Step 1: Create the scan repository​

  1. Open the template repository apisec-inc/apisec-surface-scan-template and click Use this template › Create a new repository.
  2. Set Owner to your organization and give the repository a name, such as apisec-surface-scan.
  3. Open the new repository's Actions tab and confirm that the APIsec Surface org scan workflow is listed. If Actions are disabled for the repository, enable them.

The workflow scans the organization that owns the repository, so you don't need to change the workflow file.

To use an existing repository instead, add the file templates/apisec-scan.yml to it as .github/workflows/apisec-scan.yml. The template repository contains exactly this file.

The rest of this guide calls this repository the scan repository.

note

If your organization's Actions policy allows only workflows from your own organization, the reusable workflow can't run. Use the standalone workflow instead. See If your organization blocks external workflows.

Step 2: Configure secrets and variables​

You can add secrets and variables at the organization level or on the scan repository. These steps use the organization level. Open your organization's Settings › Secrets and variables › Actions. For each secret and variable you add, set Repository access so that it includes the scan repository.

  1. On the Secrets tab, add two secrets:

    SecretValue
    APISEC_TOKENAn APIsec Personal Access Token with the Create App and Get App scopes, created by a user whose platform role is ADMIN or USER. A token from a Viewer is rejected. Create it on the API Credentials page.
    ORG_READ_TOKENA GitHub fine-grained personal access token with your organization as Resource owner. Grant Contents: Read-only and Metadata: Read-only.

  2. On the Variables tab, add three variables:

    VariableValue
    APISEC_ENDPOINThttps://api.apisecapps.com
    APISEC_REGISTRATION_CODEYour registration code, in the format ###-###
    APISEC_LLM_ENRICHMENT0. This keeps your source code on your runners. If this variable is not set, the workflow also sends selected source snippets to APIsec for LLM-assisted route discovery.

  3. Add any optional variables you need. See Optional variables.

The Repository access setting of ORG_READ_TOKEN decides which private repositories the workflow scans. Without ORG_READ_TOKEN, the workflow scans only public repositories.

note

Fine-grained tokens belong to the GitHub user who creates them, and they expire. If your organization uses a service account for automation, create the token from that account. Renew the token before its expiration date, or the nightly run fails. If your organization requires approval for fine-grained tokens, an owner must approve the token before the workflow can use it.

Optional variables​

VariableDefaultValue
ORCHESTRATOR_RUNNERGitHub-hosted (ubuntu-latest)self-hosted runs every job on runners with both the self-hosted and apisec labels. Any other value uses GitHub-hosted runners.
THROUGHPUT2The maximum number of repositories scanned in parallel, from 1 to 10.
TARGET_ORGThe organization that owns the scan repositoryThe login of a different organization to scan. ORG_READ_TOKEN must then have that organization as Resource owner.
EXCLUDE_REPOSNoneA comma-separated list of repositories to skip, as owner/repo.
APISEC_CLI_VERSION1.1.0A different version of the Surface CLI (apisec-surface).

Step 3: Run the workflow​

To start a run, open the scan repository and go to Actions › APIsec Surface org scan › Run workflow. The runner pool comes from ORCHESTRATOR_RUNNER for both manual and nightly runs.

InputDefaultEffect
dry_runfalseAnalyzes repositories but uploads nothing, and doesn't record them as scanned.
force_full_scanfalseIgnores the scan record and scans every repository.
max_repos0The maximum number of repositories in this run. 0 means no limit.

Start with two runs:

  1. Dry run. Set dry_run to true and max_repos to 5. The workflow lists repositories and analyzes up to five of them, but uploads nothing. Confirm that every job completes. Nothing appears in APIsec after a dry run.
  2. First full run. Run the workflow with the default inputs. It scans every repository, up to 256 per run, starting with the most recently pushed, and uploads the results.

The workflow also runs every night at 02:00 UTC. The schedule is active as soon as the workflow file is in the repository, so add the secrets and variables before the first nightly run. To change the schedule, edit the cron line in the scan repository's workflow file. Only one run happens at a time. A new run waits until the current run finishes.

note

Each run scans at most 256 repositories, whatever the value of max_repos. When a run reaches this cap, the Plan job log shows Reached cap of N repos; re-run for the remainder. Run the workflow again to scan the rest. The workflow skips the repositories it already scanned.

For a first scan of a large organization, set max_repos to a lower value, such as 100. Run the workflow again until the Plan job log shows Scan plan: 0 repo(s).

Verify​

  • The workflow run finishes with every job green.
  • In Applications, each discovered API appears as an application with its endpoints.
  • Run the workflow again before anyone pushes to a repository. The Plan job log shows Scan plan: 0 repo(s), and the workflow skips the scan jobs. This confirms that incremental scanning works.

How runs choose repositories​

Each run compares every repository's last push time with the time of its last successful scan. It scans only the repositories pushed to since then. The workflow keeps this record as an Actions artifact named push-ts-store in the scan repository. Each run that scans a repository refreshes the record, and GitHub keeps it for 90 days. If the record has expired or can't be restored, the run logs a warning and scans every repository.

A run always skips these repositories:

  • Archived and disabled repositories
  • Empty repositories, with no commits
  • The scan repository itself
  • The repositories in EXCLUDE_REPOS

When a scan finds no API routes in a repository, the workflow uploads nothing for it, and no application is created for it.

When a repository's scan fails, the workflow doesn't record that repository, so the next run retries it.

APIsec matches each application to its repository by the repository's GitHub ID. Later runs update the same application instead of creating a duplicate.

Control updates​

The workflow file calls apisec-inc/apisec-orchestrator/.github/workflows/scan.yml@v1. The v1 tag moves to each new 1.x release, so you get fixes automatically. Breaking changes ship as v2, which you adopt by editing the tag.

To stay on one release, replace @v1 with a release tag, such as @v1.0.1.

If your organization blocks external workflows​

If your organization's Actions policy blocks workflows from apisec-inc, replace the contents of .github/workflows/apisec-scan.yml with templates/apisec-scan-standalone.yml. Keep only one of the two workflows in the repository. It runs the same jobs inside your repository. Each run installs the latest 1.x release of the apisec-orgscan package from PyPI, so you still get fixes without editing the file.

The standalone workflow uses the same secrets and variables. Its Run workflow form adds two inputs:

InputEffect
throughputOverrides THROUGHPUT for this run.
runnergithub-hosted or self-hosted for this run. Leave it blank to use ORCHESTRATOR_RUNNER.

Move from the archive version​

If you set up an earlier version from apisec-orchestrator.zip, move to the new workflow in one of two ways. Either way, APIsec matches applications by GitHub repository ID, so neither way creates duplicates.

  • Keep your repository. In your existing orchestrator repository, replace .github/workflows/orchestrator.yml with templates/apisec-scan.yml, and delete the scripts/ folder. Your secrets, variables (including APISEC_ENDPOINT), and scan record carry over, so the next run doesn't rescan repositories that haven't changed. Add the APISEC_REGISTRATION_CODE and APISEC_LLM_ENRICHMENT variables from Step 2 if you don't have them yet.
  • Start fresh. Delete the old orchestrator repository and follow this guide from Step 1. The first run scans every repository once.

The variables ORCHESTRATOR_RUNNER and THROUGHPUT keep working. The default runner pool is now GitHub-hosted. If you use self-hosted runners, set ORCHESTRATOR_RUNNER to self-hosted. The new workflow doesn't use APISEC_SERVER_URL or the Variables: Read-only permission on ORG_READ_TOKEN.

If APIsec created applications from OpenAPI specifications for your repositories before you used the orchestrator, ask APIsec support to link those applications to their repositories before your first full scan.

Optional: Show the repository inventory in APIsec​

The scan workflow doesn't use or need a GitHub App. Add the APIsec Surface Discovery GitHub App only if you want a live inventory of your organization's repositories in the APIsec console. The app has read-only access to repository metadata and no access to code.

Create the GitHub App​

  1. In GitHub, open your organization's Settings › Developer settings › GitHub Apps and click New GitHub App.

  2. Fill in the General fields:

    FieldValue
    GitHub App nameAPIsec Surface Discovery
    Homepage URLhttps://apisec.ai
    Callback URLLeave empty.
    Webhook › ActiveSelected
    Webhook URLhttps://api.apisecapps.com/v1/integrations/github/webhook
    Webhook secretA random string. Generate one with the command after this table.
    SSL verificationEnabled
    openssl rand -hex 32

  3. Under Repository permissions, set Metadata to Read-only. Leave every other permission at No access.

  4. Under Subscribe to events, select Repository, Installation, and Installation repositories.

  5. Under Where can this GitHub App be installed?, select Only on this account, then click Create GitHub App.

  6. On the app's General page, copy the App ID.

  7. Under Private keys, click Generate a private key. GitHub downloads a .pem file.

tip

Store the webhook secret and the .pem file in your password manager now. You paste both into APIsec when you connect the app, and GitHub doesn't show the webhook secret again.

Install the app on your organization​

  1. On the app's settings page, click Install App, then click Install next to your organization.
  2. Choose All repositories, or Only select repositories to start with a subset. You can change this selection later.
  3. After installation, copy the Installation ID from the end of the page URL. For example, https://github.com/organizations/example-org/settings/installations/12345678 has the Installation ID 12345678.

The repositories you select here appear in the APIsec inventory. The repositories that get scanned are set separately, by ORG_READ_TOKEN.

Connect the app in APIsec​

  1. In the APIsec console, click Integrations.

  2. On the GitHub App tile, click View Inventories, then create a new connection.

  3. Fill in the connection form:

    FieldValue
    NameAny unique name, such as APIsec Surface Discovery
    GitHub OrganizationYour organization's login, such as example-org
    App IDThe App ID from the app's General page
    Installation IDThe Installation ID from the installation page URL
    Private KeyThe full contents of the .pem file, including the BEGIN and END lines
    Webhook SecretThe webhook secret you generated

  4. Leave Make connection private cleared. It restricts the connection to your user only.

  5. Click Test Connection. When the test succeeds, click Save.

  6. Open the connection and click Inventory. Your repositories appear with their visibility, default branch, and an APIsec column that shows Application exists or No application. The first sync runs in the background and usually finishes within a few minutes.

After a scan run, the repositories that have an application show Application exists. To test the lifecycle webhooks, archive or rename a test repository in GitHub. The inventory usually shows the change within a few minutes.

Troubleshoot​

SymptomCauseFix
A scan job fails with Set the APISEC_ENDPOINT org variable to your APIsec API URL.APISEC_ENDPOINT is missing, or the scan repository can't access itAdd the variable and include the scan repository in its Repository access.
A scan job fails with SSL certificate verify failedAPISEC_ENDPOINT points at the wrong hostSet it to https://api.apisecapps.com.
The run fails with Set the APISEC_TOKEN secret, or run with dry_run.APISEC_TOKEN is missing, or the scan repository can't access itAdd the secret and include the scan repository in its Repository access.
A scan job fails with HTTP 403APISEC_TOKEN lacks Create App or Get App, or its creator has the Viewer roleCreate a new token with both scopes from an ADMIN or USER account.
The run logs ORG_READ_TOKEN is not set; only public repos and this repo can be scanned.ORG_READ_TOKEN is missing, or the scan repository can't access itAdd the secret and include the scan repository in its Repository access.
The run fails with TARGET_ORG is empty; refusing to guess which org to scan.The workflow couldn't determine which organization to scanSet the TARGET_ORG variable to your organization's login.
A scan job fails with <path> was not uploaded (exit N); it is retried next run.The analysis or upload of that repository failedCheck the job log for the cause. The next run retries the repository.
Jobs stay queued with "Waiting for a runner"ORCHESTRATOR_RUNNER is self-hosted, and no runner has the apisec labelAdd the apisec label to your runners. To use GitHub-hosted runners instead, delete ORCHESTRATOR_RUNNER.
The run is green, but no application appearsThe run was a dry runRun the workflow again with dry_run set to false.
A repository is never scannedThe repository is outside the Repository access of ORG_READ_TOKEN, is archived or empty, or is in EXCLUDE_REPOSIn the token's settings in GitHub, add the repository to Repository access, or remove it from EXCLUDE_REPOS.
A repository isn't scanned againNobody pushed to it since its last successful scanRun the workflow with force_full_scan set to true to scan every repository.
A run scans every repository and logs that every repo counts as changedThe scan record expired, because no run scanned anything for 90 days, or the record couldn't be restoredNo action needed. This run saves a new record, and later runs scan only changed repositories.
Test Connection fails with "GitHub credentials validation failed"The App ID, Installation ID, or private key is wrongCopy each value again from GitHub. Paste the whole .pem file.
Saving the connection fails with "already connected under a different connection"Another APIsec connection already uses this installationEdit the existing connection instead.
The inventory doesn't reflect repository changesWebhook deliveries failIn the GitHub App's Advanced › Recent Deliveries, a 403 means the webhook secrets don't match, and a 404 means the webhook URL is wrong.