Source Code Repository Scanning & Rating
This page contains an explanation of our rating criteria for repositories.
- 1.What is being tested?
The test examines whether a project has unaddressed vulnerabilities within its codebase or dependencies, utilizing the OSV (Open Source Vulnerabilities) service. Examples of vulnerabilities include:
- SQL injection vulnerabilities, enable an attacker to execute arbitrary SQL queries, leading to unauthorized access or data manipulation.
- Cross-site scripting (XSS) vulnerabilities, where an attacker could inject malicious scripts into web pages viewed by other users, potentially compromising their data or hijacking their sessions.
- Path traversal vulnerabilities may allow attackers to access restricted directories and files, potentially exposing sensitive data.
- Insecure deserialization vulnerabilities, permit attackers to execute malicious code by manipulating serialized objects.
- 2.What are the benefits of having that respective aspect covered?
Addressing vulnerabilities provides several advantages:
- Enhances overall security, reducing the likelihood of successful attacks.
- Demonstrates the project's commitment to maintaining secure and up-to-date software.
- Builds trust among users and stakeholders, ensuring their data and systems are protected.
- Minimizes potential legal and reputational risks associated with security breaches.
- 3.What problems can occur if that aspect is neglected?
If this aspect is neglected, the risk potential is high:
- Ignoring known vulnerabilities could lead to exploitation by attackers, causing data breaches, unauthorized access to sensitive information, or system downtime.
- These issues can severely impact your project's reputation, user trust, and potentially result in financial losses or legal consequences.
- Compromised system integrity, making it challenging to recover from security incidents or prevent future attacks.
- 1.What is being tested?
The test examines if a project utilizes a dependency update tool to mitigate the risk of known vulnerabilities. These tools automate the process of updating dependencies by identifying outdated or insecure requirements and opening pull requests to update them. Examples of dependency update tools include:
- Dependabot: Automatically creates pull requests to update dependencies in various programming languages.
- Renovatebot: A versatile and configurable dependency update tool that supports multiple platforms and languages.
- Sonatype Lift: A tool that offers comprehensive software supply chain analysis, including dependency updates.
- 2.What are the benefits of having that respective aspect covered?
The benefits of utilizing a dependency update tool are:
- Improved security through timely updates to dependencies, reducing the likelihood of known vulnerability exploitation.
- Streamlined and automated dependency management, saving developers time and effort.
- Increased user trust and project reputation by ensuring software remains up-to-date and secure.
- 3.What problems can occur if that aspect is neglected?
If this aspect is neglected, the risk potential is high.
- Increased susceptibility to attacks that exploit known flaws in outdated dependencies.
- Higher maintenance burden for developers, who must manually track and update dependencies.
- Potential loss of user trust and damage to the project's reputation if security issues arise from outdated dependencies.
- 1.What is being tested?
The test aims to evaluate the project's maintenance status, considering factors like commit frequency and issue activity.
Key criteria assessed are:
- If there's at least one commit per week over the previous 90 days, the project receives the highest score.
- If there's activity on issues involving collaborators, members, or owners of the project, a partial score is given.
- 2.What are the benefits of having that respective aspect covered?
Ensuring active maintenance brings several benefits:
- Higher likelihood of patches and fixes for discovered vulnerabilities.
- More timely updates of dependencies, reducing exposure to known security flaws.
- Increased user confidence in the project's stability and ongoing support.
- Adaptation to evolving technology trends and changing requirements.
- 3.What problems can occur if that aspect is neglected?
Neglecting active maintenance may result in the following problems:
- Unpatched vulnerabilities, exposing the project to potential attacks.
- Outdated dependencies, increasing susceptibility to known flaws.
- Declining user trust and adoption due to perceived project abandonment.
- Reduced compatibility with new technologies and industry standards.
However, it is essential to note that some smaller utility functions or projects may not need active maintenance, as their functionality is inherently stable and not subject to frequent updates [4].
- 1.What is being tested?
The test assesses whether a project has published a security policy, typically in the form of a SECURITY.md file.
Criteria include:
- Presence of valid contact information for vulnerability reporting (e.g., email address or http/https address).
- Inclusion of free form text beyond just contact information.
- Specific text addressing vulnerability and disclosure practices, expectations, and/or timelines.
- 2.What are the benefits of having that respective aspect covered?
- Provides a clear communication channel for users to report vulnerabilities securely.
- Establishes expectations for vulnerability handling and disclosure practices.
- Fosters trust and confidence among users and stakeholders in the project's security posture.
- 3.What problems can occur if that aspect is neglected?
Ignoring the need for a security policy may lead to:
- Difficulty for users and researchers to report vulnerabilities, increasing the risk of public exposure or exploitation.
- Confusion and miscommunication about vulnerability handling and disclosure practices.
- Erosion of trust and confidence in the project's security stance.
- 1.What is being tested?
The test evaluates whether a project performs CI tests before merging pull requests, with a primary focus on GitHub repositories. CI tests help detect errors or vulnerabilities early on, leading to better code quality. The check works by looking for a set of CI-system names in GitHub CheckRuns and Statuses among the recent commits (~30). A CI-system is considered well-known if its name contains any of the following: appveyor, buildkite, circleci, e2e, github-actions, jenkins, mergeable, test, travis-ci. Examples of CI systems include:
- CircleCI: A platform for continuous integration and delivery, automating the software development process.
- Travis-CI: A hosted service for continuous integration that builds and tests projects hosted on GitHub.
- GitHub Actions: A platform for automating workflows, including CI testing, directly within GitHub repositories.
- 2.What are the benefits of having that respective aspect covered?
- Early identification and resolution of vulnerabilities and errors
- Improved code quality and maintainability
- Enhanced collaboration among developers, leading to more stable code
- 3.What problems can occur if that aspect is neglected?
If this aspect is neglected, the risk potential is low.
- Increased likelihood of unknown vulnerabilities
- Slower debugging and resolution of issues
- Weaker collaboration among developers
- 1.What is being tested?
The test assesses whether a project uses fuzzing to identify vulnerabilities in the code. Fuzzing involves feeding unexpected or random data into a program to expose bugs. The evaluation checks for the following:
- Inclusion of the repository name in the OSS-Fuzz project list
- Deployment of ClusterFuzzLite in the repository
- Presence of user-defined language-specified fuzzing functions (currently supports Go fuzzing only)
- Existence of a OneFuzz integration detection file
Examples of fuzzing tools and services include:
- 2.What are the benefits of having that respective aspect covered?
- Early detection and resolution of vulnerabilities in the code
- Improved code stability and robustness against unexpected inputs
- Enhanced overall security of the project
- 3.What problems can occur if that aspect is neglected?
If this aspect is neglected, the risk potential is medium.
- Increased likelihood of unidentified vulnerabilities in the code
- Higher susceptibility to attacks exploiting discovered flaws
- Weaker overall security of the project
- 1.What is being tested?
The test evaluates whether a project uses Static Application Security Testing (SAST), also known as static code analysis. SAST involves analyzing the source code before the application is run to prevent known classes of bugs from being introduced into the codebase. The checks look for known GitHub apps such as CodeQL (github-code-scanning) or SonarCloud in the recent (~30) merged PRs or the use of "github/codeql-action" in a GitHub workflow. The test also checks for the deprecated LGTM service until its forthcoming shutdown.
Examples of SAST tools include:
- 2.What are the benefits of having that respective aspect covered?
- Identification and prevention of known classes of bugs in the codebase
- Early detection and resolution of security vulnerabilities
- Improved overall code quality and security posture
- 3.What problems can occur if that aspect is neglected?
If this aspect is neglected, the risk potential is medium.
- Increased likelihood of undiscovered bugs in the code that can damage functionality leads to downtime etc.
- Higher susceptibility to attacks exploiting identified vulnerabilities
- Reduced code quality and overall security of the project
- 1.What is being tested?
The test assesses whether the project has generated executable (binary) artifacts in the source repository. Including such executables increases user risk as they often directly use them, leading to various issues. However, the following types of files are allowed:
- Files that are simultaneously reviewable source code and executables (e.g., files used by some interpretive systems, such as operating system shells)
- Source code generated by other tools, like bison, yacc, flex, and lex, since they are easier to review and pose a lower risk
- Generated documentation, as it is intended for human consumption and doesn't pose the same level of risk as non-reviewable code
Examples of generated executable artifacts include:
- C/C++ generated machine code
- Java .class files
- Python .pyc files
- Minified JavaScript
- 2.What are the benefits of having that respective aspect covered?
- Ensures code reviewability, reducing the likelihood of obsolete or malicious executables
- Promotes the practice of building from source, enhancing transparency and security
- Preserves the executable generation process, avoiding atrophied build systems
- 3.What problems can occur if that aspect is neglected?
If this aspect is neglected, the risk potential is high.
- Introduction of non-reviewable code, increasing vulnerability to obsolete or malicious executables
- Weakening of the executable generation process, potentially leading to an inability to create working executables
- Erosion of user trust due to the inclusion of non-transparent binaries
- 1.What is being tested?
The test assesses if a project's default and release branches have branch protection settings in place on GitHub. These settings enforce specific workflows for branches, such as requiring review or passing status checks before acceptance into a main branch, or preventing the rewriting of public history. The aspects being tested include:
- Require code review
- Prevent force push
- Require status checks
This test has tiered scoring, and each tier must be fully satisfied to achieve points at the next tier. The scoring system includes:
Tier 1 Requirements (3/10 points):
- Prevent force push
- Prevent branch deletion
- For administrators: Do not allow bypassing the above settings
Tier 2 Requirements (6/10 points):
- Required reviewers >=1
- For administrators: Last push review
- For administrators: Strict status checks (require branches to be up-to-date before merging)
Tier 3 Requirements (8/10 points):
- Status checks defined
Tier 4 Requirements (9/10 points):
- Required reviewers >= 2
Tier 5 Requirements (10/10 points):
- For administrators: Dismiss stale reviews
- For administrators: Require CODEOWNER review
Note: Allowed cases are when rules need to be suspended, such as when a past commit contains illegal content, making it necessary to use force push for rewriting history.
- 2.What are the benefits of having that respective aspect covered?
Benefits of implementing branch protection include:
- Reduced risk of compromised contributors injecting malicious code
- Increased chance of detecting and fixing unintentional vulnerabilities before accepting changes
- Prevention of public history rewriting without external notice
- Assurance that all required CI tests are met before changes are accepted
- 3.What problems can occur if that aspect is neglected?
If this aspect is neglected, the risk potential is High.
- Increased vulnerability to intentional malicious code injection
- Unintentional vulnerabilities could go undetected and unfixed
- Unregulated rewriting of public history
- Bypassing of essential CI tests
- 1.What is being tested?
This test evaluates if a project's GitHub Action workflows contain dangerous code patterns that could compromise the repository. The following patterns are examined:
- Untrusted Code Checkout: Identifying the misuse of potentially dangerous triggers, such as using pull_request_target or workflow_run workflow triggers along with an explicit pull request checkout. The test checks if these triggers are used unsafely, like in cases where PR authors may compromise the repository using build scripts controlled by them or reading tokens in memory.
- Script Injection with Untrusted Context Variables: Detecting inline scripts in a workflow that could execute untrusted input from context variables, which could lead to the execution of malicious commands and scripts. The test identifies if attackers add malicious content to specific GitHub context variables deemed untrusted, such as github.event.issue.title, which should not flow directly into executable code.
- 2.What are the benefits of having that respective aspect covered?
Benefits of avoiding dangerous code patterns in GitHub Action workflows include:
- Reduced risk of repository compromise due to untrusted code checkouts
- Prevention of script injections with untrusted context variables
- Improved overall security of the project and its workflows
- Assurance that workflows operate as intended without interference from malicious actors
- 3.What problems can occur if that aspect is neglected?
If this aspect is neglected, the risk potential is Very High:
- Vulnerability to repository compromise due to untrusted code checkouts or malicious build scripts
- Execution of malicious commands and scripts through script injections
- Unauthorized access to sensitive information, such as repository secrets
- Potential damage to the project's reputation and trustworthiness
- 1.What is being tested?
This test evaluates whether a project requires code review before merging pull requests, by examining the following technical aspects:
- Approvals on GitHub for the most recent changes (last ~30 commits)
- If the merger is different from the committer (implicit review)
- Checks for reviews using Prow (labels "lgtm" or "approved") and Gerrit ("Reviewed-on" and "Reviewed-by")
- Examination of recent changes solely attributed to bot activity (e.g., dependabot, renovatebot, or custom bots)
The scoring is leveled and is deducted based on specific conditions:
- 3 points deducted if any bot-originated changes are unreviewed
- 7 points deducted if a single human change is unreviewed
- Additional 3 points deducted if multiple human changes are unreviewed
- 2.What are the benefits of having that respective aspect covered?
Benefits of requiring code reviews include:
- Improved code quality by detecting and fixing unintentional vulnerabilities before merging[3]
- Deterrence or detection of attackers attempting to insert malicious code[4]
- Increased collaboration among contributors
- Enhanced trust in the project's codebase
- 3.What problems can occur if that aspect is neglected?
If code reviews are neglected, the risk potential is High:
- Introduction of unintentional vulnerabilities into the codebase
- Increased risk of injection of malicious code by attackers
- Lowered overall code quality and maintainability
- Damaged project reputation and trustworthiness
- 1.What is being tested?
This test evaluates whether a project has recent contributors from multiple organizations, based on the following technical aspects:
- At least three different companies represented in the last 30 commits
- Each contributor from those companies must have at least five commits within the last 30 commits
Please note that this check is currently limited to repositories hosted on GitHub and does not support other source hosting repositories.
- 2.What are the benefits of having that respective aspect covered?
The benefits of having contributors from multiple organizations include:
- Greater trust in the code due to involvement from diverse perspectives and interests
- Higher likelihood of attracting and retaining talented contributors
- Increased stability and sustainability of the project
- 3.What problems can occur if that aspect is neglected?
If a project has a lower number of trusted code reviewers, the risk potential is considered Low:
- Reduced diversity of perspectives, which may limit the project's ability to innovate and address a wide range of use cases
- Potentially slower growth and development due to a limited pool of contributors
- Greater vulnerability to disruptions if key contributors leave or become unavailable
- 1.What is being tested? This test evaluates whether a project pins its dependencies during the build and release process, based on the following technical aspects:
- Identifying unpinned dependencies in Dockerfiles, such as using "FROM node:latest" instead of "FROM node:14.17.5"
- Checking for unpinned dependencies in shell scripts, such as using "wget https://example.com/package.tar.gz" instead of "wget https://example.com/package-v1.0.0.tar.gz"
- Assessing GitHub workflows for unpinned dependencies, for example, using "actions/checkout@v2" instead of "actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f"
- Special consideration for Go modules, treating full semantic versions as pinned due to the Go tool's verification of downloaded content against hashes when first downloaded
Please note that this check is currently limited to repositories hosted on GitHub and does not support other source hosting repositories.
- 2.What are the benefits of having that respective aspect covered? Pinning dependencies offers several security benefits:
- Ensuring that checking and deployment are conducted with the same software, reducing deployment risks, simplifying debugging, and enabling reproducibility[5]
- Helping to mitigate compromised dependencies from undermining the security of the project[6]
- 3.What problems can occur if that aspect is neglected? If dependencies are not pinned, the risk potential is considered Medium:
- Possible compromised dependencies affecting the project
- Inconsistent software versions during checking and deployment, causing debugging and reproducibility issues
- Increased vulnerability to dependency confusion or substitution attacks
- 1.What is being tested? This test evaluates a project's automated workflow tokens, ensuring they are read-only by default. It examines these technical aspects:
- Reviewing each workflow's YAML file for top-level read-only permissions definitions and run-level required write permissions[1]
- Assessing write permissions that may pose security risks and reduce the project's score:
- Small reductions:
statuses: Attacker may change pre-submit checks, merging PRchecks: Attacker may remove pre-submit checks, introducing bugsecurity-events: Attacker may access vulnerability reports pre-patch; exception for recognized SARIF results upload actiondeployments: Attacker may trigger VM runs, charging repo owner, or remotely trigger unsanitized code/location variables
- Large reductions:
contents: Attacker may commit unreviewed code; exception for recognized packaging action/commandpackages: Attacker may publish packages; exception for recognized packaging action/commandactions: Attacker may steal GitHub secrets via approval-required action
Note that this check applies only to GitHub repositories and doesn't support other source hosting repositories.
- 2.What are the benefits of having that respective aspect covered? By ensuring tokens are read-only, projects reap these security benefits:
- Adherence to the least privilege principle, minimizing harm from compromised tokens[2]
- Reduced risk of attackers using write access tokens to insert malicious code into the project[2]
- 3.What problems can occur if that aspect is neglected? Neglecting read-only token permissions raises the risk potential to High:
- Greater susceptibility to malicious code additions from compromised tokens with write access
- Weaker security posture due to disregard for the least privilege principle
- 1.What is being tested? This test assesses whether a project is published as a package. It examines these technical aspects:
- Identifying GitHub packaging workflows and language-specific GitHub Actions that upload the package to a corresponding hub (e.g., Npm)
- Acknowledging various packaging methods: language-level package managers, operating system platforms, and container images
Please note this check is limited to GitHub repositories and doesn't support other source hosting repositories.
- 2.What are the benefits of having that respective aspect covered? Packaging a project grants users several advantages:
- Simplified download, installation, update, and uninstallation process via a package manager
- Easy access to security patches through updates, ensuring enhanced security
- 3.What problems can occur if that aspect is neglected? Neglecting proper packaging raises the risk potential to Medium:
- Users may miss essential security updates
- Increased difficulty in managing the software lifecycle
Keep in mind that a low assessment result isn't definitive proof of risk, as an automated tool may not detect all packaging methods.
- 1.What is being tested? This test determines if a project cryptographically signs release artifacts. It examines the following technical aspects:
- Searching for signature-related filenames (*.minisig, *.asc (pgp), *.sig, *.sign, *.intoto.jsonl) in the project's last five release assets
Please note this check is limited to GitHub repositories and doesn't support other source hosting repositories.
- 2.What are the benefits of having that respective aspect covered? Signed releases offer critical security advantages:
- Ensuring the provenance of release artifacts, confirming their origin and integrity
- Providing a safeguard against the installation of malicious releases
- 3.What problems can occur if that aspect is neglected? If release artifacts aren't cryptographically signed, the risk potential is High:
- Increased possibility of installing malicious releases
- Lack of certainty regarding the origin and integrity of release artifacts
Keep in mind that the test doesn't verify the signatures themselves. The assessment result should be considered within this context.