Data Formats

sgraph supports multiple data formats, each optimized for different use cases. This guide explains when and how to use each format.

Overview

Format	Use Case	File Size	Performance	Human Readable
XML	Large models, production	Compact	Very Fast	Moderate
Deps	Scripting, simple analysis	Small	Fast	Very High
JSON	Web applications	Medium	Moderate	High
GraphML	Graph visualization tools	Large	Moderate	Low

XML Format

The XML format is the primary format for sgraph, designed for performance and scalability.

Structure

<model version="2.1">
  <elements t="architecture">
    <e n="root_element">
      <e n="child_element" i="2">
        <r r="3" t="relationship_type" />
      </e>
      <e i="3" n="target_element">
      </e>
    </e>
  </elements>
</model>

Key Features

Integer References: Elements use integer IDs (i attribute) for efficient relationships
Compact Representation: Minimal XML overhead
Hierarchical Structure: Nested elements represent containment
Relationships: <r> tags define associations between elements
Attributes: Custom attributes stored as XML attributes

Attributes Reference

Element Attributes (`<e>`)

n - Name of the element
i - Unique integer identifier
t - Type of the element
Custom attributes as needed

Relationship Attributes (`<r>`)

r - Reference to target element ID
t - Type of relationship
Custom attributes for relationship metadata

Example: C Project Structure

<model version="2.1">
  <elements t="c_project">
    <e n="nginx">
      <e n="src">
        <e n="core">
          <e n="nginx.c" t="source_file">
            <r r="2" t="includes" />
            <r r="3" t="includes" />
          </e>
          <e i="2" n="nginx.h" t="header_file">
          </e>
          <e i="3" n="config.h" t="header_file">
          </e>
        </e>
      </e>
    </e>
  </elements>
</model>

Working with XML

from sgraph import SGraph
from sgraph.modelapi import ModelApi

# Create and export to XML
model = SGraph(SElement(None, ''))
# ... build your model ...
model.to_xml('output.xml')

# Load from XML
api = ModelApi(filepath='model.xml')
elements = api.getAllElements()

Deps Format

The Deps format is a simple, line-based text format perfect for scripting and quick analysis.

Structure

source_path:target_path:relationship_type
source_path:target_path:relationship_type:attribute1=value1,attribute2=value2

Examples

Basic Dependencies

/nginx/src/core/nginx.c:/nginx/src/core/nginx.h:includes
/nginx/src/core/nginx.c:/nginx/src/core/config.h:includes
/myapp/views.py:/myapp/models.py:imports
/myapp/views.py:/myapp/utils.py:imports

With Attributes

/api/user.py:/api/auth.py:imports:frequency=high,critical=true
/frontend/login.js:/api/auth.py:calls:method=POST,async=true

Hierarchical Elements Only

When no relationships exist, the format shows the hierarchical structure:

/nginx
/nginx/src
/nginx/src/core
/nginx/src/core/nginx.c
/nginx/src/core/nginx.h

Working with Deps Format

from sgraph import SGraph
from sgraph.converters.deps_to_xml import DepsToXml
from sgraph.converters.xml_to_deps import XmlToDeps

# Export to Deps
model.to_deps('dependencies.txt')

# Convert between formats
deps_to_xml = DepsToXml()
deps_to_xml.convert('dependencies.txt', 'model.xml')

xml_to_deps = XmlToDeps()
xml_to_deps.convert('model.xml', 'dependencies.txt')

Command Line Usage

# Convert XML to Deps
python -m sgraph.converters.xml_to_deps model.xml output.deps

# Convert Deps to XML  
python -m sgraph.converters.deps_to_xml dependencies.txt model.xml

# Filter deps files
grep "\.py:" dependencies.txt > python_deps.txt

JSON Format

JSON format provides a web-friendly representation of sgraph models.

Structure

{
  "model_version": "2.1",
  "root": {
    "name": "root",
    "type": "root",
    "children": [
      {
        "name": "module1",
        "type": "module",
        "attributes": {
          "language": "python",
          "lines": 150
        },
        "children": [],
        "relationships": [
          {
            "target_path": "/module2",
            "type": "imports",
            "attributes": {}
          }
        ]
      }
    ]
  }
}

Working with JSON

from sgraph.converters.sgraph_json import SGraphJson
from sgraph.converters.xml_to_json import XmlToJson

# Convert to JSON
converter = XmlToJson()
converter.convert('model.xml', 'model.json')

# Load JSON in web applications
import json
with open('model.json', 'r') as f:
    model_data = json.load(f)

GraphML Format

GraphML is a standard XML format for graphs, supported by many visualization tools.

Features

Compatible with Gephi, yEd, Cytoscape
Rich metadata support
Standard format for graph exchange

Working with GraphML

from sgraph.converters.xml_to_graphml import XmlToGraphMl

# Convert to GraphML
converter = XmlToGraphMl()
converter.convert('model.xml', 'graph.graphml')

# Import into visualization tools:
# - Gephi: File > Open > graph.graphml
# - yEd: File > Open > graph.graphml  
# - Cytoscape: File > Import > Network from File

CycloneDX SBOM Format

sgraph.converters.sbom_cyclonedx_generator emits CycloneDX 1.7 documents. It can produce a single SBOM for the whole model, one SBOM per element at a chosen tree depth (--level), or one for a named element (--element-path).

# One SBOM per repository, for a model whose repositories sit at depth 3
python -m sgraph.converters.sbom_cyclonedx_generator model.xml sboms.json --level 3

The transitive dependency closure

By default a document lists the 3rd-party packages the analyzed code itself declares. A package that only another package depends on — the resolved closure a lockfile records — is left out, even when the model holds those package-to-package edges.

--transitive-externals follows them, so the whole closure reaches the document. It is opt-in: the closure multiplies component counts, and every existing consumer of the default output keeps receiving exactly what it received before. --max-depth N caps how deep the walk goes.

# Direct dependencies plus everything they pull in, no deeper than two hops
python -m sgraph.converters.sbom_cyclonedx_generator model.xml sboms.json \
    --level 3 --transitive-externals --max-depth 2

Both options require --level or --element-path; the single-SBOM mode does not accept them. --max-depth requires --transitive-externals, and must be 1 or greater: a smaller cap excludes every component the walk could emit, so it is refused rather than rounded up to the shallowest level.

Field Meaning

properties[dependencyDepth] Package hops between the component and the analyzed code: 1 for a package the code declares, 2 for one that package pulls in, and so on. When a package is reachable by several routes, the shortest is reported — including routes through the internal elements a --transitive document inlines, so the depth a component publishes always agrees with where the dependencies section of the same document places it.

Field	Meaning
`properties[dependencyDepth]`	Package hops between the component and the analyzed code: `1` for a package the code declares, `2` for one that package pulls in, and so on. When a package is reachable by several routes, the shortest is reported — including routes through the internal elements a `--transitive` document inlines, so the depth a component publishes always agrees with where the `dependencies` section of the same document places it.

The property is present on every 3rd-party component of a --transitive-externals document, depth 1 included, and absent from every component of a default one. So an absent property means the document makes no depth claim — never “this package is direct”.
Only edges that mean package depends on package are followed (the manifest and lockfile deptypes). Code-level edges between externals are not: they describe how code is written, not what a manifest declares, and following them would list packages the project never depends on.
A development-section declaration is followed exactly like a production one. Whether such a package should be scoped differently in the document is a separate question this option does not answer.
The option is inert on a model that holds no package-to-package edges the converter recognises: the components are the same, only the depth property is added. What a document contains depends on what the analyzers stored, not on the flag alone. The recognised deptypes are the manifest and lockfile ones — packagejson, packagelock, pip, package_reference, nuget and pubspec, each also in its dev_-prefixed form. Anything else between two externals is skipped.
An empty closure says so. When a document follows no package-to-package edge at all while edges between external elements were skipped for their deptype, one line naming those deptypes is written to stderr. Otherwise an unrecognised deptype and a model with no closure at all produce the same document, and nothing distinguishes them.

Which repository’s edges a document follows

The External subtree is project-wide: every repository’s resolved tree lands in the same External/<ecosystem> elements and shares the versioned ones. A package-to-package edge on its own therefore says nothing about which repository’s manifest declared it, and a naive closure walks a sibling’s edges — putting packages in a document at versions that repository does not install. Measured on two real repositories in one estate, 6 % and 4 % of the closure was another repository’s.

The analyzers record the declaring scope on each such edge, as the model path of the directory whose manifest declared it (the repository root for the ordinary lockfile that sits there). Where several manifests declare the same edge, every scope is recorded — the paths joined by //, a sequence that cannot occur inside a single model path.

An edge is followed when its declaring scope and the element the document is rooted at lie on the same root-to-leaf line: the scope is that element, an ancestor of it, or a descendant of it. Not merely “inside it” — a lockfile sits at a repository root while a directory-level document is rooted below it, so an “inside” test would empty the closure of every directory-level document.

An edge with no declaring scope is followed. Every model stored before the attribute existed carries none, and the pip and NuGet analyzers record none today. Absence means unknown provenance, and unknown provenance keeps the behaviour that was there before; reading it as “skip” would silently empty those closures.

With the closure, dependencies becomes a graph rather than a flat list: every package that pulls in another gets an entry of its own, so a consumer can trace which package introduced an exposure instead of only learning that it is present.

"dependencies": [
  { "ref": "repoa", "dependsOn": ["pkg:npm/express@4.18.2"] },
  { "ref": "pkg:npm/express@4.18.2", "dependsOn": ["pkg:npm/body-parser@1.20.1"] },
  { "ref": "pkg:npm/body-parser@1.20.1", "dependsOn": ["pkg:npm/qs@6.11.0"] }
]

A package reached only through another one is listed under that package, not under the element. Listing it under the element would assert a direct dependency no manifest declares, and contradict the dependencyDepth the same document publishes for it.
A package that is both declared and pulled in appears under both.
Every ref resolves within the document, except urn:cdx: BOM-Links, which name another document by design.
Without --transitive-externals there are no package-to-package hops, so the section keeps the single flat entry it has always had.

Internal packages

A dependency that resolves to another element of the same estate is a third category: not the document’s own subject, and not a 3rd-party package. It appears as a component marked softagram:internal, alongside the urn:cdx: BOM-Link that has always been in dependsOn.

{ "bom-ref": "repob",
  "type": "library",
  "name": "ui-lib",
  "version": "2.1.0",
  "purl": "pkg:generic/ui-lib@2.1.0",
  "group": "/OrgName/GroupA",
  "properties": [
    { "name": "softagram:internal", "value": "true" },
    { "name": "softagram:packageName", "value": "ui-lib" },
    { "name": "softagram:packageEcosystem", "value": "npm" },
    { "name": "softagram:elementPath", "value": "/OrgName/GroupA/repoB" }
  ],
  "externalReferences": [
    { "url": "urn:cdx:b02cf884-4fe6-5d96-8bab-a649ae9844b2/1", "type": "bom" },
    { "url": "https://example.org/org/repoB.git", "type": "vcs" }
  ] }

Field	Meaning
`name` / `version` / `purl`	The package the element publishes, when the model says unambiguously which one that is. Otherwise the element’s own name, an empty version and an empty purl.
`properties[softagram:internal]`	Always `"true"` on these components.
`properties[softagram:packageName]`	The published package name, the same one spliced into the purl.
`properties[softagram:packageEcosystem]`	The ecosystem the package is published in (`npm`, `pypi`, …). Absent when the model does not name one; it decides no part of the purl.
`externalReferences[type=bom]`	BOM-Link to that element’s own standalone document.

The purl type is generic, never the ecosystem’s own type. pkg:npm/<name>@<version> would assert an identity in the public npm registry. Either the name is not published there, in which case the npm type buys nothing, or it is and belongs to someone else, in which case the component silently inherits a stranger’s advisories. The ecosystem is published as a property instead, so nothing is lost but the false claim.
The dependency is emitted both as this component and as a urn:cdx: BOM-Link in dependsOn. The link federates to the element’s own document for a consumer that follows links across uploads; the component is what a consumer that does not follow them can see at all.
Only direct internal dependencies are inlined into a default document. --transitive inlines the whole chain of them, together with the 3rd-party components of every link.
In a default document the component appears only when the element publishes a package identity; without one, dependsOn carries the BOM-Link alone, exactly as before. A component named after a repository or a directory, with no version and no purl, is the very shape the missing-identifier problem is about, and until an analyzer stamps the identity attributes every such component on an existing model would come out that way. So the default document changes for a consumer only once the row can carry real coordinates.
--transitive deliberately does not take that rule. Inlining internal elements is what that option has always done and its consumers already receive those rows; identity improves them where it exists, but its absence must not delete a dependency a consumer can see today. The asymmetry is intentional, not an inconsistency.
How many of these a document holds scales with the granularity you select. They are not a fixed overhead: they are one component per element the chosen element directly depends on, so the count follows the --level (or --element-path) you pass. A repository-level export usually gains a handful. A directory-level export splits the same estate into far more, and finer, elements, so dependencies that were internal to one repository become cross-element ones — on a large repository a single directory-level document has been measured gaining several hundred. That is the number of dependencies that directory genuinely has, not a closure being walked; if it is more than you want, export at a coarser level.
An element that publishes several packages with nothing to choose between them is given no identity at all: it keeps the element name and carries no purl key. No coordinates are invented. (Before 1.13.0 the key was present and empty, which CycloneDX types as an iri-reference and the empty string is not one.)
Components describing internal packages carry no dependencyDepth. That property counts package hops through the 3rd-party closure, and an element of the estate is not one.

What the document does not cover

A component list cannot say what is missing from it. An external the analyzer saw and the generator could not identify leaves no trace, so a consumer cannot tell “this estate depends on nothing else” from “we could not identify the rest”.

--coverage says it, in single-SBOM mode:

python -m sgraph.converters.sbom_cyclonedx_generator model.xml sbom.json --coverage

It adds two things. Four metadata.properties counting what was and was not identified under External — components emitted, not a package, version unknown by design, could not identify — and one compositions entry stating whether the third-party assembly is complete:

"compositions": [{ "aggregate": "incomplete", "assemblies": ["/Org"] }]

`aggregate`	When
`incomplete`	Externals were seen that could not be identified.
`unknown`	Everything seen was identified. Whether everything was seen is a different question.
`not_specified`	No `External` subtree at all, so there is no basis for a claim.

complete is never emitted. CycloneDX defines it as “no further relationships … are KNOWN to exist”. The report proves only that every element the walk saw was classified, never that the analyzer saw everything — an analyzer that was not run leaves no evidence that it was missing. Claiming completeness makes a consumer stop looking, so the strongest honest claim is unknown, which the specification defines as a best-effort whose completeness is inconclusive.
Single-SBOM mode only. The report is model-wide, so attaching it to a --level or --element-path document would claim that that document’s assembly is incomplete because some other subtree’s is. --coverage combined with either is refused rather than emitted with a caveat.
Opt-in. Without the flag the document is exactly what earlier releases produced.
The four counts are properties rather than part of the composition because a composition holds aggregate, assemblies, dependencies and vulnerabilities and nothing else. Completeness fits the standard; a ten-category taxonomy with counts and samples does not.

Where an element lives

Every component that describes a model element — the metadata component of each document, and every internal component — publishes its position in the model:

{ "bom-ref": "repoa",
  "name": "repoA",
  "group": "/OrgName/GroupA",
  "type": "application",
  "version": "",
  "purl": "",
  "externalReferences": [
    { "url": "https://example.org/org/repoA.git", "type": "vcs" }
  ],
  "properties": [
    { "name": "softagram:elementPath", "value": "/OrgName/GroupA/repoA" }
  ] }

Field	Meaning
`group`	The full path of the parent element, not just its name. Omitted for a top-level element, which has no parent path.
`properties[softagram:elementPath]`	The element’s own full path. A property rather than a field because the CycloneDX component schema sets `additionalProperties: false`.
`externalReferences[type=vcs]`	The `repo_url` of the element, or of the nearest ancestor carrying a non-blank one. Absent when no ancestor has one — never a placeholder.

Components describing 3rd-party packages carry none of these. Their identity is the purl.

Guarantees

Every properties[].value is a JSON string, including the ones that carry a number. CycloneDX types the field as a string in every spec version this converter emits, and sets additionalProperties: false, so a numeric value is not merely untidy: a validating consumer rejects the whole document rather than the one field. Counts such as indirectExposureCount are therefore published as "3", not 3, and a consumer reading them numerically has to parse them — the same as for dependencyDepth.
deterministic_serial(elementPath) == serialNumber. The published path is the exact string the serial is derived from, so a consumer can verify a document’s identity without the model.
group + '/' + name == elementPath below the top level.
Two repositories that share a name under different groups are distinguished by group and elementPath. They are not reliably distinguished by bom-ref, whose collision suffix (repoa, repoa-2) depends on traversal order and can change between model generations.

Caveats

The first path segment is the estate root and is not stable. It changes when the estate is renamed or restructured. Read it from the path rather than hardcoding it.
group holds a path, not a package namespace. The CycloneDX specification suggests avoiding special characters in group and shows package coordinates such as org.apache.commons. A model group is a tree location, and the slash-delimited path is what makes two identically named groups distinguishable, so this converter prefers precision over that convention. Tools that render group as a package coordinate will show the path.
purl and version are empty on the metadata component, and on an internal component whose element publishes no unambiguous package. A repository has no package identity and no version of its own; a path is not a valid purl and is deliberately not placed there. An internal component whose element does publish one carries that package’s coordinates — see Internal packages.
The vcs reference is inherited by proximity. A repository with no remote of its own, under a group that has one, reports the group’s URL.

Format Comparison

Performance Benchmarks

Format	1K Elements	100K Elements	1M Elements
XML Load	10ms	500ms	5s
Deps Load	5ms	200ms	2s
JSON Load	15ms	800ms	8s
GraphML Load	20ms	1.2s	12s

File Size Comparison

For a typical software project with 10K elements:

Format	File Size	Compression Ratio
XML	2.5 MB	1.0x (baseline)
Deps	800 KB	3.1x smaller
JSON	4.2 MB	1.7x larger
GraphML	8.1 MB	3.2x larger

Best Practices

Choosing the Right Format

Use XML when:

Working with large models (>10K elements)
Need maximum performance
Building production systems
Preserving all metadata and attributes

Use Deps when:

Simple dependency analysis
Scripting and automation
Human-readable output needed
Working with shell tools (grep, awk, etc.)

Use JSON when:

Building web applications
Need JavaScript compatibility
Creating REST APIs
Moderate-sized models (<50K elements)

Use GraphML when:

Importing into visualization tools
Sharing with researchers
Need standards compliance
One-time analysis tasks

Performance Optimization

For Large Models

# Use XML format for storage
model.to_xml('large_model.xml')

# Load with ModelApi for efficient querying
api = ModelApi(filepath='large_model.xml')

# Use specific queries instead of loading all elements
functions = api.getElementsByType('function')
specific_elements = api.getElementsByName('main')

For Streaming Processing

# Process deps format line by line for very large files
def process_large_deps_file(filepath):
    with open(filepath, 'r') as f:
        for line in f:
            if ':' in line:
                parts = line.strip().split(':')
                source, target, rel_type = parts[:3]
                # Process dependency
                yield source, target, rel_type

Memory Management

# For very large models, use streaming
from sgraph.loader.modelloader import ModelLoader

# Load incrementally
loader = ModelLoader()
for element_batch in loader.load_streaming('huge_model.xml', batch_size=1000):
    # Process batch
    process_elements(element_batch)

Migration Between Formats

Preserving Metadata

When converting between formats, be aware of metadata preservation:

From → To	Elements	Relationships	Attributes	Performance
XML → Deps	✅ Paths only	✅	⚠️ Limited	Fast
XML → JSON	✅	✅	✅	Medium
XML → GraphML	✅	✅	✅	Slow
Deps → XML	✅	✅	⚠️ Limited	Fast

Batch Conversion

import os
from sgraph.converters.xml_to_deps import XmlToDeps

def convert_project_models(input_dir, output_dir):
    """Convert all XML models to Deps format"""
    converter = XmlToDeps()
    
    for filename in os.listdir(input_dir):
        if filename.endswith('.xml'):
            input_path = os.path.join(input_dir, filename)
            output_path = os.path.join(output_dir, filename.replace('.xml', '.deps'))
            
            print(f"Converting {filename}...")
            converter.convert(input_path, output_path)

# Usage
convert_project_models('models/', 'deps_output/')

Advanced Features

Custom Attributes in XML

<e n="MyClass" t="class" visibility="public" complexity="high" loc="250">
  <r r="2" t="inherits" strength="strong" />
</e>

Relationship Attributes in Deps

/src/main.py:/src/utils.py:imports:frequency=10,last_used=2023-12-01

Nested Attributes in JSON

{
  "name": "MyFunction",
  "attributes": {
    "metrics": {
      "complexity": 15,
      "lines": 45,
      "parameters": 3
    },
    "metadata": {
      "author": "developer",
      "last_modified": "2023-12-01"
    }
  }
}

Understanding these formats allows you to choose the right tool for each task and integrate sgraph into your development workflow effectively!

Data Formats

Overview

XML Format

Structure

Key Features

Attributes Reference

Element Attributes (<e>)

Relationship Attributes (<r>)

Example: C Project Structure

Working with XML

Deps Format

Structure

Examples

Basic Dependencies

With Attributes

Hierarchical Elements Only

Working with Deps Format

Command Line Usage

JSON Format

Structure

Working with JSON

GraphML Format

Features

Working with GraphML

CycloneDX SBOM Format

The transitive dependency closure

Which repository’s edges a document follows

Internal packages

What the document does not cover

Where an element lives

Guarantees

Caveats

Format Comparison

Performance Benchmarks

File Size Comparison

Best Practices

Choosing the Right Format

Performance Optimization

For Large Models

For Streaming Processing

Memory Management

Migration Between Formats

Preserving Metadata

Batch Conversion

Advanced Features

Custom Attributes in XML

Relationship Attributes in Deps

Nested Attributes in JSON

Element Attributes (`<e>`)

Relationship Attributes (`<r>`)