Automating OpenSSF Scorecard Security issues with Qodo CLI Agentic Workflow

Getting a security report for security vulnerabilities and misconfiguration issues of your GitHub project is a good start but can we leverage AI to also remediate all of these issues automatically through agentic workflow? Yes and I will show you how I do that with Qodo CLI Agentic Workflow.

The OpenSSF Scorecard is a project from the Open Source Security Foundation (OpenSSF) that provides a set of automated checks to assess the security posture of open source projects. Running the Scorecard is a nice way to get a quick overview of outstanding security gaps but can I help developers out by automating the process of running the Scorecard AND fixing all the security issues it finds? Well, yes! This is the mission that I was set out to accomplish and I want to show you how I did that with Qodo Command, a new agentic CLI workflow tool from the cool folks at Qodo.

The magic? we started with mere 2.6 points (out of 10) Scorecard and ended up with 7.4 points after running the Qodo Command agentic workflow to fix all the security issues automatically:

Ready for the full ride? Let’s go!

What is OpenSSF Scorecard and how to run it?

I already gave you a short brief on what Scorecard is about. Basically it runs a bunch of checks against your GitHub project - both your code base (more specifically, your GitHub Actions workflows) and your GitHub repository settings - and then it scores your project based on how well you’re doing on security.

You can run the Scorecard on your GitHub project by using the scorecard command, or use the ossf/scorecard GitHub Action, or even run a partial check from the OpenSSF Scorecard website. Why partial? because access to your repository settings, to vet them, is required and from the website you can only run the checks on the repo from a viewer perspective, not as an admin.

Anyways, a quick and easy to get started is:

1. Generate a GitHub Personal Access Token (PAT) with repo scope.
2. Install the scorecard CLI tool by following the instructions on the OpenSSF Scorecard GitHub repository or website
3. Run the Scorecard against your GitHub repository:

GITHUB_AUTH_TOKEN=1234 scorecard --repo "https://github.com/lirantal/npq"

Once you run the Scorecard, it will, well, generate a… drum roll … scorecard report! Surprise, surprise. Anyways, here is an example of the report you get from the CLI output if you run it on a sort of brand-new typical project without too much security hardening:

As you can see, a scorecard result of 2.6 (out of 10), pretty poor.

Now what do you do? How many developers are actually going to run the tool in the first place, let alone, read the results and start remediating the security issues one-by-one? And what happens when you need to do this at scale to a bunch of repositories? Ahh yes, frustration and despair.

Let’s automate this!

What is Qodo Command?

I know you heard all of these pitches before about Cline, Cursor, GitHub Copilot and on and on. But this time I’m going to show you Qodo because they have a unique “agent” approach to their CLI tool. And yes, as you assumed, Qodo is also in the AI-first DevTool business which means that they are building tools to help developers automate their workflows using AI. Nothing new in 2025 :D

So, Qodo Command is one of their tools that is specifically designed to help developers create agentic workflows that run on the command-line. If you’ve heard or used Claude Code then this is similar but there’s a cool twist to it.

What’s unique about Qodo Command is that it allows you to create workflows through an agent.yaml file where you define the behavior, the MCP (Model Context Protocol) servers available to the agent, and even a sort of test criteria (kind of like evals for LLMs) to ensure that the agent behaves as expected, and then also set the schema for the agent’s output.

There is a lot in it and I won’t cover it end-to-end but we’ll focus on a setup and the specific agentic workflow use-case that I created to automate the OpenSSF Scorecard security issues remediation.

Setting up Qodo Command

To get started with Qodo Command, you need to install it. You can do that by following the instructions on their GitHub repository or the Qodo Command website but it’s basically installing an npm package:

npm install -g @qodo/command

Once you have Qodo Command installed, you should authenticate:

qodo login

Follow the whole browser login flow and then you should be good to go. Back to the terminal, you can run:

qodo

And get started to prompt it and ask it questions like any other LLM or agentic workflow tool you’re used to.

Setting up the OpenSSF Scorecard agentic fixer

Now to the cool part, automating the whole OpenSSF Scorecard security issues. Here is how I went about doing this. Remember, that you execute the agentic workflow from the CLI using the qodo command, so you’ll also going to need a specific setup to make this work from your own terminal and the requirements prescription looks like this:

• Create a GitHub Personal Access Token (PAT) with repo scope. I already mentioned this above but if you didn’t yet do it, now is the time.
• Get Docker running. We will need that to run a local GitHub MCP Server.
• Install the GitHub gh CLI tool. This is required to run the gh commands from the agentic workflow. (The reason is that the AI agent will perform some tasks via the GitHub MCP Server and other tasks via the gh CLI tool)
• Install the scorecard CLI tool. This is required to run the OpenSSF Scorecard checks from the agentic workflow.
• Install the, no just kidding, no more installs. Unless you haven’t yet installed Qodo Command, in which case: npm install -g @qodo/command

Now that you have all the prerequisites, for the agent to drive work through, we need to actually define the agent.

Defining the agent means giving it instructions, telling it which MCP Servers are available to it, and so on. To do that, we’re going to create a file called agent.yaml in whatever directory you want to run the agent from. For example, I created a directory called openssf-scorecard-agent and then created the agent.yaml file in there. Note, it is likely that the agent will clone your target repository to this directory so make sure you have enough space on your disk and that there’s no other directory with the same name in this same location.

So here’s the full contents of Qodo Command’s agent.yaml file that I created and after you see it, I’ll briefly explain some parts of it so it would make more sense, but it should be pretty self-explanatory:

version: "1.0"

commands:
  openssf-scorecard-fixer:
    description: "Automatically fix and address issues raised by the OpenSSF Scorecard tool to increase security of a code repository on GitHub"
    
    arguments:
      - name: repo
        type: string
        required: true
        description: "The GitHub repository to analyze and fix issues for (full URL)"
      
      - name: create_pr
        type: boolean
        required: false
        default: true
        description: "Whether to create a pull request with the fixes (default: true)"
      
      - name: enable_branch_protection
        type: boolean
        required: false
        default: true
        description: "Whether to enable branch protection rules (default: true)"
      
      - name: fix_vulnerabilities
        type: boolean
        required: false
        default: true
        description: "Whether to automatically fix known vulnerabilities (default: true)"
      
      
      - name: base_branch
        type: string
        required: false
        default: "main"
        description: "Base branch to create feature branches from (default: main)"
      
      - name: branch_name
        type: string
        required: false
        default: "security/openssf-scorecard-fixes"
        description: "Name for the feature branch (default: security/openssf-scorecard-fixes)"
    
    available_tools:
      - git
      - filesystem
      - shell
      - github

    execution_strategy: plan
    
    mcpServers: |
      {
          "shell": {
            "command": "uvx",
            "args": [
              "mcp-shell-server"
            ],
            "env": {
              "ALLOW_COMMANDS": "scorecard,docker,env,ls,cat,pwd,rg,wc,touch,find,mkdir,rm,cp,mv,npm,npx,jest,mocha,ts-node,tsc,node,jq,echo,test,diff,sed,awk,git,cd,exit,yarn,grep,gh,base64,curl,python3,python,pip,pip3,which,whoami,id,uname,date,head,tail,sort,uniq,tr,cut,xargs"
            }
          },
          "fetch": {
            "command": "docker",
            "args": ["run", "-i", "--rm", "mcp/fetch"]
          },
          "github": {
            "command": "docker",
            "args": [
              "run",
              "-i",
              "--rm",
              "-e",
              "GITHUB_PERSONAL_ACCESS_TOKEN",
              "ghcr.io/github/github-mcp-server"
            ],
            "env": {
              "GITHUB_PERSONAL_ACCESS_TOKEN": "<YOR_TOKEN>"
            }
          }
      }
    
    instructions: |
      You are an expert product security engineer with a focus on improving the security posture of open source projects. Your task is to automatically address and fix issues raised by the OpenSSF Scorecard tool, which identifies security vulnerabilities and best practices in code repositories. You should analyze each issue, leverage tools at your disposal, determine the appropriate action, and execute it to enhance the security of the codebase.

      ## Process:

      1. **Your Tool Box**

         - You can find the GitHub Personal Access token in the environment variable GITHUB_AUTH_TOKEN
         - For all GitHub operations, prefer using the GitHub CLI (`gh`) commands over the GitHub MCP server when encountering API limitations
         - If GitHub MCP server operations fail, clone the repository locally using `git clone` and work with local files
         - To run the OpenSSF Scorecard tool, use: `scorecard --repo=<INSERT REPO>` (replace with the repository from the `repo` argument)
         - You can get information from URLs using the fetch tool
         - For base64 decoding, use: `echo "<base64_string>" | base64 -d` or Node.js: `node -e "console.log(Buffer.from('<base64>', 'base64').toString())"`
         - Always verify GitHub CLI authentication with `gh auth status` before attempting operations
         - Use `gh api` for direct GitHub API calls when the MCP server fails
         - Respect the command arguments: create_pr, enable_branch_protection, fix_vulnerabilities, base_branch, branch_name

      2. **Issue Analysis**
         - Run the OpenSSF Scorecard tool on the provided repository URL to identify issues
         - Parse the output to extract issues and their details
         - For each issue, gather all the information you can about it so that you can address it effectively
         - If you encounter "Command not allowed" errors, try alternative approaches:
           - Use GitHub CLI (`gh`) instead of direct API calls
           - Clone repository locally if remote operations fail
           - Use Node.js for data processing if shell utilities are restricted

      3. **Action Execution**

         For Each Issue:
         - Execute the most appropriate fix to address the issue
         - If GitHub MCP server operations fail, use this fallback strategy:
           1. Clone the repository locally: `git clone <repo_url>`
           2. Create a new branch: `git checkout -b <branch_name>` (use the branch_name argument)
           3. Make necessary changes to local files
           4. Commit changes: `git add . && git commit -m "security: fix OpenSSF Scorecard issues"`
           5. Push branch: `git push -u origin <branch_name>`
           6. Create PR (if create_pr=true): `gh pr create --title "security: fix OpenSSF Scorecard issues" --body "<description>"`
         - For branch protection (if enable_branch_protection=true), use: `gh api --method PUT repos/<owner>/<repo>/branches/<base_branch>/protection --field <settings>`
         - For vulnerability fixes (if fix_vulnerabilities=true), run: `npm audit fix` or equivalent for the package manager
         - Always clean up temporary files created during the process
         - Common security fixes to implement:
           - Add explicit permissions to all GitHub Actions workflows
           - Pin GitHub Actions to specific commit hashes
           - Add SAST scanning (CodeQL, Semgrep)
           - Add dependency review workflows
           - Enable Dependabot or similar automated dependency updates
         - In a separate and dedicated new Pull Request, add fuzzing workflows for appropriate projects and add an example fuzzy testing JavaScript file as most appropriate to the project, as part of the pull request changes suggested.

      4. **Error Handling**

         - If unable to address a fix for the issue, post an explanation of what you tried to do and why it was not possible
         - For complex issues beyond scope, suggest breaking into smaller tasks
         - Common error patterns and solutions:
           - "Command not allowed": Add the command to ALLOW_COMMANDS or use alternative approaches
           - "MCP error -32603": Switch to GitHub CLI or local repository operations
           - "Branch not protected": Use GitHub CLI to enable branch protection
           - Base64 decoding issues: Use Node.js or alternative decoding methods
         - Always attempt multiple approaches before declaring a task impossible
         - Document workarounds used for future reference

      ## Important Guidelines:
      - Always maintain code quality and follow existing patterns
      - If you are introducing a breaking change always provide clear notes about it
      - Keep comments professional and helpful
      - Respect repository contribution guidelines if present
      - Never close the issue yourself
      - When working with GitHub Actions, always pin actions to specific commit hashes for security
      - Add explicit permissions to all workflow files to follow principle of least privilege
      - Test fixes locally when possible before pushing changes
      - Clean up any temporary files created during the process
      - Verify improvements by running OpenSSF Scorecard again after implementing fixes

      ## Output:

      After handling all the issues, provide a summary of all actions detected by OpenSSF Scorecard and all the actions you've taken to address them, and the results.
      
      Include in your summary:
      - Initial OpenSSF Scorecard score
      - Final OpenSSF Scorecard score (run the tool again after fixes)
      - List of issues addressed with specific actions taken
      - Any issues that could not be resolved and why
      - Links to created pull requests (if any)
      - Recommendations for ongoing security maintenance

• First off, we version the agent.
• The commands section defines a command name openssf-scorecard-fixer which is what we invoke with the qodo CLI and all of its arguments that we can pass to it.
• The available_tools section lists the tools that the agent can use, including the ability to run shell commands and interact with GitHub and of course the scorecard command which is the main tool we want to use.
• The mcpServers section defines the MCP Servers that are available to the agent and have been declared in the available_tools section.
• The instructions section is where we define the agent’s behavior, what it should do, how it should handle issues, and so on. This is where we tell the agent to run the OpenSSF Scorecard tool, analyze the issues, and then fix them automatically.

Note, I did need to tweak the agent instructions a bit to avoid putting work into criteria that the scorecard tool doesn’t support very well, like fuzzing for JavaScript projects is incredibly limited and even if added to the project it isn’t detected very well.

Now we can run the agentic workflow and let it do its magic.

Running the Agentic workflow

Of course, replace the GITHUB_AUTH_TOKEN with your own GitHub Personal Access Token (PAT) and the repo argument with the full URL of the GitHub repository you want to analyze and fix issues for, and then spin up the agentic workflow by running the following command:

GITHUB_AUTH_TOKEN=1234 qodo openssf-scorecard-fixer --set repo="https://github.com/lirantal/hello-world-js"

Results be like:

Most of the issues can be fixed by the agent and others are just sometimes not fit. For example, maybe there’s no actual release process in place and it doesn’t make sense because this isn’t a project that delivers assets.

Scorecard Fixer agent results

Ok, so how well did the agent do to fix all the security issues reported by the OpenSSF Scorecard? Let’s compare the results of the first run.

As you remember, the scorecard command reported a score of 2.6 out of 10 across 18 issues. Partial summary notes from the qodo command output:

Perfect! I have successfully implemented comprehensive security improvements for the hello-world-js repository. Here's a complete summary of all actions taken:

## OpenSSF Scorecard Security Improvements Summary

### Initial Assessment
- **Initial Score**: 2.6/10
- **Current Score**: 2.9/10 (limited by token permissions for some checks)
- **Expected Final Score**: 8+/10 (after PR merges and workflow execution)

As you can see, the current score is 2.9 out of 10 which is just a slight improvement but most of the work that the agent did to configure DevSecOps pipelines for security hardening is staged in a pull request that still needs to be merged, hence there’s an expected final score as well.

Here are the pull requests that the agent opened to fix the security issues, I’m going to merge them and we’ll see the changes right after:

But even before merging the pull requests, look at how detailed the agent was in its summary of actions taken as part of the changes introduced to files in this pull request:

You can browse to the pull request in my test repository here to see the full details of the changes made by the agent: security: implement OpenSSF Scorecard security improvements #1.

Oh wait, it looks like I can’t even merge the pull request. Why? because the Qodo agent configured the repository to prevent merging pull requests without reviews, which is a good security hardening practice. This is already taking effect and preventing me or anyone else (like a rogue contributor or malicious actor who took over someone’s account) from merging the pull request without a review:

The Qodo agent also configured branch protection rules to prevent force-pushing to the main branch, which is another good security hardening practice:

The full summary of actions taken by the agent is as follows:

### Security Issues Identified and Addressed

#### ✅ **SAST (Static Application Security Testing)** - 0/10 → Expected 8+/10
**Actions Taken:**
- Added **CodeQL workflow** with pinned actions (commit hash: 692973e3d937129bcbf40652eb9f2f61becf3332)
- Added **Semgrep workflow** for additional SAST coverage
- Configured **explicit permissions** following principle of least privilege
- Set up **scheduled scanning** (weekly) and **PR-triggered scans**

#### ✅ **Security Policy** - 0/10 → Expected 10/10
**Actions Taken:**
- Created comprehensive **SECURITY.md** file with:
  - Vulnerability reporting guidelines
  - Response timeline commitments
  - Supported versions matrix
  - Private disclosure process
  - Security best practices

#### ✅ **Dependency Update Tool** - 0/10 → Expected 10/10
**Actions Taken:**
- Configured **Dependabot** for automated dependency updates
- Set up **weekly update schedule** for npm and GitHub Actions
- Added **dependency review workflow** for PR security checks
- Configured **reviewers and assignees** for dependency PRs

#### ✅ **Branch Protection** - 0/10 → Expected 10/10
**Actions Taken:**
- **Enabled branch protection** on main branch with:
  - Required pull request reviews (1 approver)
  - Dismiss stale reviews when new commits are pushed
  - Enforce admin compliance
  - Prevent force pushes and deletions
  - Strict status checks

#### ✅ **Fuzzing** - 0/10 → Expected 8+/10
**Actions Taken:**
- Created **comprehensive fuzzing workflow** with:
  - Scheduled weekly fuzzing runs
  - PR-triggered fuzzing tests
  - Timeout protection (5 minutes)
  - Artifact collection for analysis
- Implemented **extensive fuzzing test suite** covering:
  - String input validation (injection, buffer overflow)
  - Numeric input testing (overflow, division by zero)
  - Object input testing (prototype pollution, JSON safety)
  - Array input testing (memory exhaustion, boundary checks)

#### ✅ **Token Permissions** - N/A → Expected 10/10
**Actions Taken:**
- Added **explicit permissions** to all GitHub Actions workflows
- Implemented **principle of least privilege**
- Specified minimal required permissions for each job

#### ✅ **Pinned Dependencies** - N/A → Expected 10/10
**Actions Taken:**
- **Pinned all GitHub Actions** to specific commit hashes instead of tags
- Enhanced security by preventing supply chain attacks
- Used latest stable versions with security patches

### Package Security Enhancements

#### ✅ **Package.json Security**
**Actions Taken:**
- Added **security audit scripts**: `npm run security:audit`, `npm run security:audit:fix`
- Added **fuzzing scripts**: `npm run fuzz:test`, `npm run fuzz:install`
- Created **package-lock.json** for dependency integrity
- Ran **npm audit** - no vulnerabilities found

As I also requested it to report on the changes it made to the repository and other actions it took, it also provided a summary of those:

### Pull Requests Created

#### 🔗 **Main Security Improvements PR**
- **URL**: https://github.com/lirantal/hello-world-js/pull/1
- **Title**: "security: implement OpenSSF Scorecard security improvements"
- **Includes**: CodeQL, Semgrep, Dependabot, Security Policy, Package Security

#### 🔗 **Fuzzing Implementation PR**
- **URL**: https://github.com/lirantal/hello-world-js/pull/2
- **Title**: "security: add comprehensive fuzzing implementation"
- **Includes**: Fuzzing workflow, comprehensive test suite, documentation

### Files Created/Modified

#### New Security Files:
- `.github/workflows/codeql.yml` - CodeQL SAST scanning
- `.github/workflows/dependency-review.yml` - Dependency security checks
- `.github/workflows/semgrep.yml` - Semgrep SAST scanning
- `.github/workflows/fuzzing.yml` - Comprehensive fuzzing tests
- `.github/dependabot.yml` - Automated dependency updates
- `SECURITY.md` - Security policy and vulnerability reporting
- `fuzz/hello-world-fuzz.js` - Comprehensive fuzzing test implementation
- `fuzz/README.md` - Fuzzing documentation and usage guide
- `package-lock.json` - Dependency integrity

#### Modified Files:
- `package.json` - Added security and fuzzing scripts

### Security Best Practices Implemented

1. **Pinned Dependencies**: All GitHub Actions pinned to commit hashes
2. **Explicit Permissions**: Minimal permissions for all workflows
3. **Automated Security**: Continuous security scanning and dependency updates
4. **Comprehensive Testing**: Multiple SAST tools and fuzzing coverage
5. **Documentation**: Clear security policies and procedures
6. **Branch Protection**: Enforced code review and protection rules

Splendid agent work!

Now, I did have to disable branch protection rules entirely to allow myself to merge the pull request that the agent created and the dependabot PRs so that they would take effect but this was a test repository that started from scratch with just a sample Express JavaScript application to show the difference between going from bare-bones default repository setup to a fully security-hardened repository with DevSecOps pipelines in place.

Let’s now run the OpenSSF Scorecard again after merging the pull requests from the agent and see how well it did:

$ scorecard --repo="https://github.com/lirantal/hello-world-js" --show-details

Starting [License]
Starting [Security-Policy]
Starting [Binary-Artifacts]
Starting [Maintained]
Starting [Code-Review]
Starting [Dependency-Update-Tool]
Starting [Fuzzing]
Starting [SAST]
Starting [Packaging]
Starting [Contributors]
Starting [Branch-Protection]
Starting [Token-Permissions]
Starting [Pinned-Dependencies]
Starting [CII-Best-Practices]
Starting [CI-Tests]
Starting [Dangerous-Workflow]
Starting [Vulnerabilities]
Starting [Signed-Releases]
Finished [CI-Tests]
Finished [Dangerous-Workflow]
Finished [Vulnerabilities]
Finished [Signed-Releases]
Finished [License]
Finished [Security-Policy]
Finished [Binary-Artifacts]
Finished [Maintained]
Finished [Code-Review]
Finished [Dependency-Update-Tool]
Finished [Fuzzing]
Finished [SAST]
Finished [Packaging]
Finished [Contributors]
Finished [Branch-Protection]
Finished [Token-Permissions]
Finished [Pinned-Dependencies]
Finished [CII-Best-Practices]

RESULTS
-------
Aggregate score: 7.4 / 10

Hah! we scored 7.4 out of 10 with zero effort!

Even the criteria missed here that answers the 2.6 points gap isn’t a big deal. Here’s why and where the points are missing:

| 8 / 10  | Branch-Protection      | branch protection is not       | Warn: codeowners review is not required. Warn: required approving review count is 1 on 

Well I’m just one maintainer on this new repo so approval count of 1 is it. I indeed have no CODEOWNERS file, perhaps that is something that OpenSSF Scorecard should consider as a recommendation before it judges on it (it doesn’t).

Also these points we didn’t earn:

| 0 / 10  | CII-Best-Practices     | no effort to earn an OpenSSF best practices badge detected | https://github.com/ossf/scorecard/blob/ab2f6e92482462fe66246d9e32f642855a691dc1/docs/checks.md#cii-best-practices 

Well you have to sign-up manually for that, so understandable.

Next:

| 0 / 10  | Code-Review            | Found 0/3 approved changesets

Makes sense too, this is just a new repository with literally a handful of commits just so scaffold it and whatever the agent created as pull requests. All of these I had to skip approval for the sake of this demo and just merge them to show you the status. In reality, you’d be working with contributors or a team of developers and you’d be reviewing each other’s work.

Fuzzing we skipped so 0 out of 10 points there too:

| 0 / 10  | Fuzzing                | project is not fuzzed

And this is a new project so obviously we’re not going to score any “high maintenance” points for it either:

| 0 / 10  | Maintained             | project was created within the last 90 days. Please review

Summary

All in all, the Qodo Command agentic workflow proved to be a splendid work of art to create real, tangible value of 10x productivity for developers and security teams, and even for solo maintainers themselves.

It was able to run the OpenSSF Scorecard, analyze the issues, and then fix them automatically through an agentic workflow. By having access to the GitHub repository via tools and CLI that we made available to it, the agent was able to create pull requests with the fixes it found and implemented, and then also configure the repository with security hardening practices that are now in place and will continue to be enforced going forward.