been thinking about why finding training data is still such a slog in 2026. someone at berkeley recently built a tool just to search kaggle, huggingface and data . gov at the same time, because doing it by hand was too slow. the fact that this is a tool people have to build says a lot

the usual explanation is fragmentation, which is true but kind of surface level. the deeper thing is incentives. on most data marketplaces the vendor pays to be listed, and almost nobody works on commission. so the platform is optimized for whoever pays for placement, and whether the data is actually findable or usable ends up an afterthought. the money doesn't come from discovery, so discovery gets neglected

feels like it should run the other way. a request-first marketplace, where the buyer posts what they actually need and providers come back with offers. kind of like how taxis worked before the apps. you announce where you're going, drivers see it and take it if the terms work for them. the buyer sets the direction and providers respond to it

this matters more as requests get weirder. the more specialized and one-off your data ask is, the lower the odds that any single provider already has exactly that sitting on a shelf. a static catalog struggles with that kind of ask. something you can post and have people bid on fits it a lot better

genuinely curious what people here think. has anyone seen a marketplace that works request-first, or runs on commission instead of paid placement? and if you've bought data before, did you actually find it through a marketplace, or did you end up emailing providers directly because search never surfaced the right thing?

disclosure, i work at Titan Network and we're on the provider side of this, so it's something i think about a lot. not pitching anything, btw

Access vehicle specifications, history reports, images, valuations, market listings, and automotive market insights through powerful APIs.

Hello!

This dataset I have created last year has been downloaded over 20 000 times already! It's the biggest open club football dataset in the world, including over 240 000 football matches' data such as scores, stats, form, Elo, odds and more, all updated up to 09/26!

The dataset can be used for training AI models, creating visualizations, or just for personal data exploration :)

If anyone wants to dive deep into leveraging bookmaker's odds, I have also created a dataset specifically for that with 2000 matches and 41 betting markets. It's behind a paywall for a symbolic price because it supports my future work and allows for other free datasets to be updated: https://odds.adamgabor.eu/

All of the links:

Kaggle: https://www.kaggle.com/datasets/adamgbor/club-football-match-data-2000-2025/data

Github: https://github.com/xgabora/Club-Football-Match-Data

HuggingFace: https://huggingface.co/datasets/xgabora/club-football-match-data

Odds Dataset: https://odds.adamgabor.eu/

Any of y'all know where to find an oil dataset with DDRs, time series data, with good amount of completion? I tried the volve equinor one, but the formation tops for like ~70% is NULL,

Hi, I’m looking for a free dataset or API for Indian packaged food products.

Does anyone know of a good source? Thanks!

I wanted a per-product count of the seven FD&C dyes still certified for use in US food, and it turns out the Open Food Facts bulk export is enough on its own. Posting the method and the numbers because I could not find this cut published anywhere.

Source: the Open Food Facts full CSV export, https://world.openfoodfacts.org/data. I used the 2026-08-31 snapshot, 1.275 GB gzipped, 4,535,553 rows. Filter to countries_tags containing en:united-states and require a non-empty ingredients_text, which leaves 444,943 products. Then match the ingredient text against the US number form, the common chemical name and the E number for each dye.

The part I did not expect is that the naming is regular. I went in braced for the inulin problem, where a third of the products containing an ingredient never print its common name and you spend the afternoon chasing synonyms. It is not there. Of the 6,259 products matching any Red 3 form, 23 do not use the plain "Red 3" wording, which is 0.4%. Red 40 is 96 of 28,591, or 0.3%. Yellow 5 is the worst at 0.8%, because tartrazine still turns up. A regex on the number form catches essentially all of it, which is not how ingredient text usually behaves.

There is a regulatory reason for that. 21 CFR 101.22(k)(1) requires a colour additive subject to certification to be declared by the name given in the applicable regulation, and it explicitly allows dropping the "FD&C" prefix and the "No." So "Red 40" is not shorthand, it is the compliant form. Only colours not subject to certification may be declared under (k)(2) as "Artificial Color", "Artificial Color Added" or "Color Added".

That flips something I had assumed going in. 16,768 products use one of those generic phrases, and 10,571 of them also name a certified dye, so in those the generic phrase is covering something else in the same product. 3,914 use a generic phrase and name no colourant at all, and by (k)(2) whatever those are, they are not one of the seven.

Where the label does go dark is (k)(3), which says colouring added to butter, cheese and ice cream need not be declared at all. Most producers declare anyway: 27.5% of the 12,080 cheese records name a natural colourant, mostly annatto. But the exemption is real and it is invisible from the ingredient list.

On timing, FDA revoked the Red 3 authorization on 2025-01-15 under the Delaney Clause and gave food manufacturers until 2027-01-15 to reformulate, so this snapshot sits four months out from the deadline. The 6,256 products span 1,073 distinct first-brands, so it is not one catalogue duplicated.

Limitations, and the second one is large.

• Open Food Facts is crowd contributed. This is a sample of what people scanned, not the US shelf, and popular products are over-represented.
• Records go stale. Only 17.3% of the Red 3 rows were modified after the revocation date, against 43.0% of all rows. Restrict to rows touched since then and Red 3 falls from 1.41% to 0.57%. I am quoting both because neither is the honest single number: the full count includes labels that have already changed, the restricted count drops products nobody has rescanned.
• A string match cannot see inside "natural flavors" or a sub-ingredient it does not parse, so every figure here is a floor.
• I hand-read 12 random Red 3 matches looking for false positives and found none. They are all "fd&c red #3", "red 3 lake" and similar.

The gap I cannot close from this export is retail availability. Does anyone know a source that would let me weight this by what is actually stocked, rather than by whoever happened to scan it?

Disclosure per rule 1: I write an iOS app called Snack Check that reads these dyes off a barcode, which is why the parser was already sitting there. I am not linking it, and the export above is the only link in this post.

Disclosure: I built this and it is hosted on my own site, quant500.com. It is free, CC0, no account and no API key. I am posting it because the time-of-day column does not seem to exist anywhere else for free, and because the defects in it are worth more discussion than the coverage.

What it is. Every Form 8-K carrying item 2.02 filed by an S&P 500 company: 64,938 rows, one per filing, covering 63,969 distinct announcements (group by cik + announcement_date) from 808 companies, 2003-04-25 to 2026-09-01. Sixteen columns. Every row carries the accession number and a direct link to the filing on sec.gov, so any line can be checked at source.

Original source is SEC EDGAR (data.sec.gov/submissions). This is a derived file, rebuilt daily.

The column that seems to be missing elsewhere is the clock time, and with it the session the news could first be traded in: 46.4% before the open, 41.8% after the close, 11.7% during the session. 99.8% resolve to a New York time.

Four defects, because you would find them anyway:

1. The 8-K cover-page date is typed by the filer and is often wrong. Micron enters the fiscal quarter end there in 14 of its 94 announcements; a 2004 Walmart filing declares the event as happening in 2001. Where the cover date sits more than three days from the EDGAR entry, the EDGAR date is used and the row carries date_uncertain = yes. That is 3,569 rows.

2. EDGAR's acceptance timestamp ends in "Z" but is not always UTC. Measured on this file: 924 rows carry a raw hour between 00:00 and 05:59, impossible if the stamp were already Eastern because EDGAR is closed then, and 4,835 carry one between 06:00 and 09:59, impossible if it were UTC. Both conventions are genuinely present.

3. And the one I cannot fix. I resolve that convention per company, which is probably the wrong unit: the share of announcements from companies classified as "already New York" falls steadily from 24.1% in 2003 to 1.0% in 2026. A fixed company attribute should not drift like that, so the convention likely belongs to the filing agent or to the era. It is declared in the file header and unresolved. If anyone here has parsed EDGAR at scale and has seen this, I would like to know.

4. The filing does not always arrive on the day of the announcement. In 7,695 rows (11.8%) filing_date is later than announcement_date, by a single day in 6,265 of them. For those rows the session and the clock time describe the day the filing arrived, not the day the news broke. So if you group by announcement_date and read the session column without also reading filing_date, you place 11.8% of the sample in the wrong trading session. Use filing_date to know which day the session refers to.

Limits: nothing before 2003-03-28, when the 8-K had no dedicated earnings item; item 2.02 only exists from 2004-08-23, and earlier filings come in under the old item 12 and are marked rule = 12; and the session recorded is the one in which the 8-K reached EDGAR, not the one in which the press release went out, which is earlier.

CSV: https://quant500.com/api/descarga/anuncios.csv

Method and caveats: https://quant500.com/blog/2026-09-04-conjunto-de-datos-resultados

Browsable: https://quant500.com/earnings-date

CORRECTION, 2026-09-06. Two things in the text above are superseded. I am appending rather than rewriting, so anyone who read the original can see what changed.

Defect 3 was wrong about the unit, and it is now solved. I said the convention was assigned per company and that this was probably the wrong unit. It is not per company, and it is not the era either. u/TilmanAmbach compared six filings between the raw SGML header of the full submission and the submissions JSON: the raw ACCEPTANCE-DATETIME is always US/Eastern, and it is the JSON that applies two different treatments to it, converting some records to UTC (+4 summer, +5 winter) and appending a "Z" to others while leaving the Eastern clock untouched. His AA / ACT pair settles it: same year, same filer-agent prefix, opposite treatments. So it is per record, and decidable rather than inferable - the truth is readable in the SGML header. My file still takes its times from the JSON, so time_et and session are provisional until I re-ingest. That re-ingest is running now.

The 924 figure used the wrong window. u/Ian_Gow pointed out that EDGAR only accepts filings 06:00-22:00 ET (sec.gov/submit-filings), which decides which raw hours carry a signature at all. Redone properly on the 64,938 rows: 4,835 (7.4%) must be Eastern; 1,504 (2.3%), not 924, must be UTC over 23:00-03:59; 2 rows are impossible under either reading; and 58,597 (90.2%) carry no signature at all. That last number is the real headline and it is not a flattering one: nine rows in ten hold no evidence about their own convention.

I have since sampled 70 of the rows I label as already-Eastern against their raw SGML headers: 69 were right and 1 was wrong, and the wrong one had been published as 10:47 during-session when the filing was actually accepted at 06:47, before the open. Small rate, worst possible kind of error.

Both corrections are in the CSV header too, dated, with the old figures left standing and marked superseded.

Disclosure: I run genchijapan.com. This is my own dataset.

What it is

A table of the rules foreign visitors keep asking about in Japan — is Gion closed to photos, can you drink on the street in Shibuya, is there a fine for smoking in Osaka, what the tax-free refund rules actually say — with the things that are usually missing from English answers:

• level — national law / prefectural ordinance / city ordinance / district rule with no statutory force (e.g. a neighbourhood council's posted ¥10,000 "fine")
• authority — who issued it
• penalty and since — fine amount and effective date
• source — URL of the primary source (e-Gov statute text, ministry or city page) or, where the issuer has no web page, press coverage
• verified — date a human last checked the row against the source

Manners are excluded on purpose: they can't be sourced.

Honest limits

• 17 rows across 7 areas. Small. Rows are added as I verify them, not scraped.
• A few rows (mainly Gion) cite press instead of the issuing body, because the district council publishes nothing online. source_label tells you which.
• verified is a date, not a guarantee. Rules change.

Links

• GitHub (JSON + CSV + schema): https://github.com/genchijapan/japan-tourist-rules
• Live JSON, CORS-enabled: https://genchijapan.com/rules.json

CC BY 4.0. Corrections with a source URL are the most useful thing you can send me.

Recently I'm stuck hard on my research at finding extremely similar datasets, but should still be different. It seems that even datasets in the same semantic field differs drastically inside an LLM's internal representations. Any good way of finding datasets this similar?

By the way I'm especially interested in datasets that are at least 5,000 samples large, which narrows the candidates greatly.

Hey guys. I've been training reinforcement learning models and pulling my hair out trying to get clean, historical order book depth data without paying institutional API fees. I finally set up my own pipeline to capture L2 depth and imbalance for SOL. I formatted it specifically for backtesting and feature engineering. You can grab the dataset here:https://imbalancelabs.com/datasets/sol-historical-orderbook. What specific L2 features (besides standard imbalance and spread) are you feeding into your models lately?

Adaption has launched a new solution - Invent a Dataset - that allows you to to describe in a few words what you want a model to learn and it turns that objective into a structured, training-ready [Synthetic] dataset. Most synthetic-data tools automate generation only after someone has defined the schema, task distribution, and generation strategy. Invent a Dataset starts one level earlier: with the behavior the model should learn.

Hi everyone,

I'm shutting down a crypto market research / ML project that I've been running since late 2025, and I've decided to release a free 7-day sample of the market data I collected for anyone interested in quantitative research, market microstructure, backtesting or machine learning.

The sample contains 81,579 ADA/USDC market snapshots with 97 columns.

What's included

• 20 bid + 20 ask Level-2 order book levels
• Bid/ask prices and quantities
• Best bid / best ask
• Mid price and spread
• Latest spot trade price and quantity
• Trade direction
• Futures open interest
• Funding rate
• Mark price
• Order-book imbalance
• Market depth at different distances from mid price
• Liquidity wall ratio
• Book pressure

The collector runs on a nominal 5-second polling cycle. Because API calls and processing occur between snapshots, the actual median interval in this sample is approximately 6.17 seconds.

The original timestamps are preserved and the public sample has not been artificially interpolated or resampled.

Free download

Hugging Face:
rfab85/crypto-5s-market-data-adausdc-sample · Datasets at Hugging Face

Kaggle:
ADA/USDC High-Frequency Market Microstructure Data

I'd be genuinely interested to hear what people working with L2/order-book data think of the schema and what features you would derive from it.

Hi, I was looking for this dataset 'BIWI Kinect Head Pose'. But it is no longer available for the public, If anyone has the dataset downloaded please DM me.

has anyone ever needed govt datasets for a project and spent significant time finding/accessing it? or is it just me

I spent the last year gathering as much Usenet text data as possible from a large variety of resources and including from about 7000 live active servers as well to fill in gaps.

The end result is a searchable/filterable dataset acessible via web+api (free and paid options)

API : https://www.usenet-rewind.com/api-docs

Web: https://www.usenet-rewind.com

Binaries have been filtered out and range is from posts from 1981-Current and constantly populating. the site includes a histogram of distribution of message counts per year.

I welcome feedback or possible use cases. Built on Apache Solr 10, Mariadb 12 (rocksdb) and NVME disk hardware for fastest operation.

[self-promotion] [free] [paid]

Just uploaded the full 5.94 billion TikTok video dataset to Hugging Face. It’s fully open source:
https://huggingface.co/datasets/kuben-developer/tiktok-videos-4b

This dataset was collected using a TikTok mobile app reverse-engineering method I developed a few years ago. The method allowed me to extract billions of videos, profiles, comments and replies, hashtags, sounds, and more.

Full write-up and code here:
https://tiktok-api.seeksocial.io

Disclaimer: The TikTok app exposes 24 endpoints that can be accessed without a TikTok account, so the data itself is publicly accessible. But accessing it this way is probably still against TikTok’s ToS. Also, the full code is not free, I charge a small fee for access to it.

Disclosure first: the ads come from a US data center job board I run, so this is my own data and a link to my own project. Nothing is paywalled or email-gated.

What it is. Once a month I freeze every live ad on the board at one recorded UTC instant and publish the aggregates. August 2026 is the first release:

• 1,341 live ads, 62 employers, 41 states, 254 city-state pairs
• 769 ads (57.3%) carry employer-stated pay
• Two CSVs: aggregates (live jobs by state, role family and employer, with shares) and pay bands (median advertised lower and upper bound by role and pay period)
• CC BY 4.0

Zenodo, version DOI: https://doi.org/10.5281/zenodo.22261858 — the series DOI https://doi.org/10.5281/zenodo.22261857 always resolves to the newest edition.

Original source, where the method and every edition live: https://datacenterjobhub.com/data-center-hiring-index/2026/08

Method. The population is every ad live and approved at the cutoff, 2026-09-02T10:15:54Z. Pay is employer-stated only: where a job feed supplied its own estimate the ad counts as not stating pay, and ads with a floor but no ceiling are excluded from the bands. Annual and hourly are reported separately and never converted. A role x pay-period group is published only at 8 or more ranges. "Median advertised lower bound" is the median of the ranges' minimums, so it is a statistic about advertised ranges, not about earnings.

One cut that is not in the bundle yet. I ran the stated-pay rate by state against the twelve US jurisdictions that require a pay range in the advertisement itself. This was read at a later cutoff, 2026-09-03T15:10:37Z and 1,351 live ads, so it does not tie exactly to the frozen edition above:

• 86.1% of ads state pay in the 9 mandate states that have live ads; 51.8% in the other 31
• dropping the largest employer, which states pay in 174 of its 175 ads, widens that to 84.9% against 44.0%
• 13 employers run 5 or more ads on each side of the line. Compared with themselves: 93.8% in mandate states, 81.0% elsewhere. So it is not only employer mix
• of the eight largest data center markets, only California has a posting mandate. Virginia has none and states pay in 76% of 152 ads; Texas manages 41% of 340

That CSV goes into the September edition's folder so it is citable rather than just asserted here.

Limits. One board's ads, not the market: it skews to employers whose careers sites are ingested, and 62 employers is not the industry. Mandate status is as of 2026-01-30. 33 ads carry no state and sit outside the state figures. Every number above comes from a script in the repo.

Hey all, I work at CompanyData. A while back we published free business datasets for a few major cities on Datahub: https://datahub.io/@companydatadotcom (i can also share a link from our own website, just let me know in comments)

Each dataset covers companies in one city, sourced from official government registries. Current fields per company: registered address, registration number, SIC industry code, revenue, employee count, and ownership information through headquarter linkages, so you can see which companies belong to the same group. The Berlin one turned out to be the most popular by far, which we didn't expect.

No commercial angle here, downloads are free. We publish these because registry-sourced company data should be easier to get your hands on than it is.

What I'd really like feedback on is the fields themselves. Which ones actually matter for your work, and what's missing? Founding date, legal form, ownership links, NACE instead of SIC, officer counts, website domains? We decide what goes into the next batch based on what people here say, so be critical. Also open to suggestions for the next city or country.

I’m researching how to build or fine-tune an audio-to-chord-recognition engine comparable in ambition to Song Master Pro / Auralis Sound Prism.
The goal is not basic major/minor chord detection. I need reliable recognition of dense harmonic material: jazz, soul, funk, neo-soul, Brazilian music, film music, and arrangements with chords such as maj9, 6/9, m9, m11, 13, altered dominants, slash chords/inversions, secondary dominants, modal interchange, suspensions, passing harmony, etc.
Most public datasets I’ve found seem too limited: either simplified chord labels, weak annotations, or repertoire that does not really cover sophisticated harmony. In particular, I need time-aligned audio + chord labels, ideally with beat/downbeat information and a rich, consistent chord vocabulary.
My questions:
Which open datasets are genuinely useful for this level of chord-recognition work?
Are there any commercial/licensable datasets with high-quality, detailed chord annotations that can legally be used to train a model and ship it in commercial software?
Is a dataset such as iReal Pro-style chord charts, Hooktheory, Ultimate Guitar, Chordify, or similar usable in any legitimate/licensable way — or are they generally not viable due to rights and annotation quality?
For a serious model, is the realistic route to combine public datasets with a privately licensed/hand-annotated corpus? If so, roughly how many accurately annotated tracks would be needed before it becomes meaningfully good at jazz-influenced harmony?
Are there papers, benchmarks, companies, or dataset vendors I should study before spending money?
I’m specifically looking for practical, legally usable data sources—not advice to scrape chord sites. Any experience from people who have trained MIR / chord-recognition models would be very valuable.

I recently completed a dataset containing 51,080 publicly funded broadband locations in Virginia.

The project involved combining multiple public datasets, including statewide address points, administrative boundaries, broadband funding records, and service qualification results into a single address dataset.

The largest challenge was not collecting the data but reconciling it. No single source contained all funded locations. Records had to be standardized, deduplicated, geographically validated, merged across multiple funding programs, and linked back to authoritative address points while preserving provenance.

The final dataset contains:

• ADDRESS
• CITY
• COUNTY
• STATE
• ZIP
• LATITUDE
• LONGITUDE
• SOURCE_COUNTY
• FUNDING_TYPE
• SOURCE_DATASET
• VATI_INCLUDED
• BEAD_INCLUDED
• CURRENT_APB_RESULT
• APB_V2_RESULT
• APB_can_apb_service_address
• APB_lookup_status
• APB_match_quality
• APB_lookup_type
• APB_available_bundle_count
• APB_available_bundle_names
• APB_lookup_timestamp

After constructing the dataset, each address was evaluated through the public All Points Broadband ordering system and classified as:

yes
no
unknown_no_suggestion
unknown_bad_resolution
unknown_error
not_checked

Several issues emerged during processing, including duplicate addresses appearing in multiple programs, inconsistent formatting across source datasets, addresses resolving to incorrect counties, and addresses resolving to entirely different states.

Rather than discarding problematic records, unresolved locations were retained and classified separately so that every funded location remained accounted for.

The project ultimately became an exercise in address matching, GIS integration, provenance tracking, and public infrastructure data management.

Primary data sources:

Virginia Geographic Information Network (VGIN)

https://vgin.vdem.virginia.gov

Virginia statewide address points

Virginia Administrative Boundary Dataset

Virginia Telecommunication Initiative (VATI) 2022

Virginia Telecommunication Initiative (VATI) 2023

Virginia Telecommunication Initiative (VATI) 2024

Virginia Broadband Equity, Access, and Deployment (BEAD) post challenge locations

Virginia Office of Broadband

https://www.dhcd.virginia.gov/broadband

Virginia Broadband Map

https://vast.virginia.gov

Northern Shenandoah Valley Regional Commission

https://www.nsvregion.org

Rappahannock Broadband Authority Project Resources

https://rappbroadband.org/about-the-project/