Class GeoSingleCellDetectorTest

java.lang.Object

ubic.gemma.core.util.test.BaseTest5

ubic.gemma.core.loader.expression.geo.GeoSingleCellDetectorTest

@Tag("slow") @Tag("geo")
@ContextConfiguration
@ExtendWith(NetworkAvailableExtension.class)
@NetworkAvailable(url="ftp://ftp.ncbi.nlm.nih.gov/geo/series/")
public class GeoSingleCellDetectorTest
extends BaseTest5

TODO: Phase 2 — trim and cache per-test download data so the integration-tagged methods can move to fully offline @Test. SOFT files are now cached in test resources (Phase 1).

Constructor Summary

Constructors

Constructor	Description
GeoSingleCellDetectorTest()

Method Summary

Modifier and Type	Method	Description
void	setUp()
void	testDownloadSingleCellDataInCellXGene()
void	testDownloadSingleCellDataInCellXGeneWithoutACollectionId()
void	testGSE109774()	This series has numerous TAR archives with numerous small files within.
void	testGSE125708()	This series has samples lacking data.
void	testGSE132188()	This is an AnnData dataset that uses the .h5ad.h5 extension.
void	testGSE147495()	This is a case of a MEX dataset where files are located in the series.
void	testGSE148611()
void	testGSE155499()	This example has some additional supplementary materials that are inside a TAR archive that contains MEX data.
void	testGSE155695()	This dataset has technical replicates, this is not supported.
void	testGSE158184()	This is a curious example of a SOFT file that has NONE indicators in its supplementary materials.
void	testGSE159416()	This is a Loom dataset with a single file in the series.
void	testGSE162631()	This is a pathologic case with nested zip and gzipped files.
void	testGSE162807()	This is a custom H5 attachment that is unsupported.
void	testGSE165635()	The SRA metadata does not use the "TRANSCRIPTOMIC SINGLE CELL" library source, but keywords in the protocol indicate it is single-cell.
void	testGSE169285()
void	testGSE174574()	Interesting case because it uses genes.tsv instead of features.tsv.
void	testGSE178226()	MEX dataset stored in ZIP archives.
void	testGSE179516()	This is a Loom dataset with files in individual samples
void	testGSE184506()	This series has complete MEX files at the sample-level, but also provide MEX files in the series.
void	testGSE185737()	This dataset as a mix of scRNA-Seq and ATAC-Seq
void	testGSE193884()
void	testGSE196516()	This file has a large TAR attachment in its ATAC-seq samples that should be ignored.
void	testGSE199762()	This one uses non-standard file names.
void	testGSE200202()	This files produces "java.util.zip.ZipException: invalid block type" apparently...
void	testGSE200218()	This dataset does not use 'single-cell transcriptomic' as a library source, so we have to rely on keywords.
void	testGSE201032()
void	testGSE201263()	Similar case here with large TAR attachments, but in this case it has a lot of samples to skip.
void	testGSE201814()	This is a case of a MEX dataset where files are bundled in a per-sample TAR archive.
void	testGSE201814DownloadAll()
void	testGSE202051()	This dataset has multiple .h5ad supplementary files.
void	testGSE208742()	This dataset has MEX data at both series and sample-levels.
void	testGSE218621()	This is a MEX dataset with a barcode_metadata.tsv.gz file.
void	testGSE221522()
void	testGSE221593()	This AnnData file has invalid columns and numerical categorical arrays and uses a dense matrix.
void	testGSE224438()	Typical MEX format: 3 supplementary files per sample
void	testGSE225158()	AnnData (and also Seurat Disk, but the former is preferred)
void	testGSE228370()
void	testGSE235314()	This series contains large TAR attachment that should be skipped when inspecting for MEX.
void	testGSE237862()	This dataset has a mixture of Loom and MEX, the latter should take precedence because Loom is not supported yet.
void	testGSE242423()	This is a case of a MEX dataset that has matrix and barcodes at the sample-level (as we support it!), but genes at the series level.
void	testGSE244451()	This dataset as an array of ENUM, a raw.X and needs to be transposed.
void	testGSE251724()
void	testGSE254569()
void	testGSE256105()
void	testGSE274772()
void	testGSE71585()
void	testGSM4282408()	This sample has duplicated cell IDs.
void	testHasSingleCellDataInCellXGene()
void	testHasSingleCellDataInSra()
void	testIsSingleNuclei()

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

GeoSingleCellDetectorTest

public GeoSingleCellDetectorTest()

Method Details

setUp

@BeforeEach
public void setUp()
           throws IOException

Throws:

IOException

testGSE225158

@Test
@Disabled
public void testGSE225158()
                   throws IOException,
NoSingleCellDataFoundException

AnnData (and also Seurat Disk, but the former is preferred)

Throws:

IOException

NoSingleCellDataFoundException

testGSE221593

@Test
@Disabled
public void testGSE221593()
                   throws IOException,
NoSingleCellDataFoundException

This AnnData file has invalid columns and numerical categorical arrays and uses a dense matrix.

Throws:

IOException

NoSingleCellDataFoundException

testGSE221522

@Test
@Tag("integration")
@NetworkAvailable(url="ftp://ftp.ncbi.nlm.nih.gov/geo/series/")
public void testGSE221522()
                   throws IOException,
NoSingleCellDataFoundException

Throws:

IOException

NoSingleCellDataFoundException

testGSE254569

@Test
@Disabled
public void testGSE254569()
                   throws IOException,
NoSingleCellDataFoundException

Throws:

IOException

NoSingleCellDataFoundException

testGSE224438

@Test
public void testGSE224438()
                   throws IOException,
NoSingleCellDataFoundException

Typical MEX format: 3 supplementary files per sample

Throws:

IOException

NoSingleCellDataFoundException

testGSE201814

@Test
@Disabled
public void testGSE201814()
                   throws IOException,
NoSingleCellDataFoundException

This is a case of a MEX dataset where files are bundled in a per-sample TAR archive.

Throws:

IOException

NoSingleCellDataFoundException

testGSE201814DownloadAll

@Test
@Disabled("This is simply too slow to be practical.")
public void testGSE201814DownloadAll()
                              throws IOException,
NoSingleCellDataFoundException

Throws:

IOException

NoSingleCellDataFoundException

testGSE228370

@Test
public void testGSE228370()
                   throws IOException,
NoSingleCellDataFoundException

Throws:

IOException

NoSingleCellDataFoundException

testGSE174574

@Test
@Tag("integration")
@NetworkAvailable(url="ftp://ftp.ncbi.nlm.nih.gov/geo/series/")
public void testGSE174574()
                   throws IOException,
NoSingleCellDataFoundException

Interesting case because it uses genes.tsv instead of features.tsv.

We need to download it to make sure that the file is properly stored.

Throws:

IOException

NoSingleCellDataFoundException

testGSE202051

@Test
@Tag("integration")
@NetworkAvailable(url="ftp://ftp.ncbi.nlm.nih.gov/geo/series/")
public void testGSE202051()
                   throws IOException

This dataset has multiple .h5ad supplementary files. The solution is to manually pick one of them.

Throws:

IOException

testGSE200218

@Test
public void testGSE200218()
                   throws IOException,
NoSingleCellDataFoundException

This dataset does not use 'single-cell transcriptomic' as a library source, so we have to rely on keywords.

It als has a bunch of additional supplementary files.

Throws:

IOException

NoSingleCellDataFoundException

testGSE162631

@Test
public void testGSE162631()
                   throws IOException

This is a pathologic case with nested zip and gzipped files.

Throws:

IOException

testGSE148611

@Test
public void testGSE148611()
                   throws IOException

Throws:

IOException

testGSE193884

@Test
@Disabled("This is an example of a single MEX dataset at the series-level for all the samples.")
public void testGSE193884()
                   throws IOException,
NoSingleCellDataFoundException

Throws:

IOException

NoSingleCellDataFoundException

testGSE147495

@Test
@Tag("integration")
@NetworkAvailable(url="ftp://ftp.ncbi.nlm.nih.gov/geo/series/")
public void testGSE147495()
                   throws IOException,
NoSingleCellDataFoundException

This is a case of a MEX dataset where files are located in the series. We don't support downloading those, but we want to make sure we're producing a useful error message.

Throws:

IOException

NoSingleCellDataFoundException

testGSE242423

@Test
@Tag("integration")
@NetworkAvailable(url="ftp://ftp.ncbi.nlm.nih.gov/geo/series/")
public void testGSE242423()
                   throws IOException,
NoSingleCellDataFoundException

This is a case of a MEX dataset that has matrix and barcodes at the sample-level (as we support it!), but genes at the series level.

Throws:

IOException

NoSingleCellDataFoundException

testGSE132188

@Test
public void testGSE132188()
                   throws IOException,
NoSingleCellDataFoundException

This is an AnnData dataset that uses the .h5ad.h5 extension.

Throws:

IOException

NoSingleCellDataFoundException

testGSE162807

@Test
public void testGSE162807()
                   throws IOException

This is a custom H5 attachment that is unsupported.

Throws:

IOException

testGSE158184

@Test
public void testGSE158184()
                   throws IOException

This is a curious example of a SOFT file that has NONE indicators in its supplementary materials.

Throws:

IOException

testGSE185737

@Test
public void testGSE185737()
                   throws IOException,
NoSingleCellDataFoundException

This dataset as a mix of scRNA-Seq and ATAC-Seq

Throws:

IOException

NoSingleCellDataFoundException

testGSE196516

@Test
public void testGSE196516()
                   throws IOException,
NoSingleCellDataFoundException

This file has a large TAR attachment in its ATAC-seq samples that should be ignored.

Throws:

IOException

NoSingleCellDataFoundException

testGSE235314

@Test
public void testGSE235314()
                   throws IOException

This series contains large TAR attachment that should be skipped when inspecting for MEX.

Throws:

IOException

testGSE201263

@Test
public void testGSE201263()
                   throws IOException,
NoSingleCellDataFoundException

Similar case here with large TAR attachments, but in this case it has a lot of samples to skip.

Throws:

IOException

NoSingleCellDataFoundException

testGSE200202

@Test
@Disabled("This files produces \"java.util.zip.ZipException: invalid block type\" apparently.")
public void testGSE200202()
                   throws IOException,
NoSingleCellDataFoundException

This files produces "java.util.zip.ZipException: invalid block type" apparently...

Throws:

IOException

NoSingleCellDataFoundException

testGSE155499

@Test
public void testGSE155499()
                   throws IOException,
NoSingleCellDataFoundException

This example has some additional supplementary materials that are inside a TAR archive that contains MEX data.

Throws:

IOException

NoSingleCellDataFoundException

testGSE201032

@Test
public void testGSE201032()
                   throws IOException

Throws:

IOException

testGSE218621

@Test
@Tag("integration")
@NetworkAvailable(url="ftp://ftp.ncbi.nlm.nih.gov/geo/series/")
public void testGSE218621()
                   throws NoSingleCellDataFoundException,
IOException

This is a MEX dataset with a barcode_metadata.tsv.gz file.

Throws:

NoSingleCellDataFoundException

IOException

testGSE237862

@Test
public void testGSE237862()
                   throws IOException,
NoSingleCellDataFoundException

This dataset has a mixture of Loom and MEX, the latter should take precedence because Loom is not supported yet.

Throws:

IOException

NoSingleCellDataFoundException

testGSE179516

@Test
@Tag("integration")
@NetworkAvailable(url="ftp://ftp.ncbi.nlm.nih.gov/geo/series/")
public void testGSE179516()
                   throws IOException,
NoSingleCellDataFoundException

This is a Loom dataset with files in individual samples

Throws:

IOException

NoSingleCellDataFoundException

testGSE159416

@Test
@Disabled
public void testGSE159416()
                   throws IOException,
NoSingleCellDataFoundException

This is a Loom dataset with a single file in the series. We support detection and download, but not loading.

Throws:

IOException

NoSingleCellDataFoundException

testGSE178226

@Test
public void testGSE178226()
                   throws IOException,
NoSingleCellDataFoundException

MEX dataset stored in ZIP archives.

Throws:

IOException

NoSingleCellDataFoundException

testGSE199762

@Test
@Tag("integration")
@NetworkAvailable(url="ftp://ftp.ncbi.nlm.nih.gov/geo/series/")
public void testGSE199762()
                   throws IOException,
NoSingleCellDataFoundException

This one uses non-standard file names.

Throws:

IOException

NoSingleCellDataFoundException

testGSE155695

@Test
@Tag("integration")
@NetworkAvailable(url="ftp://ftp.ncbi.nlm.nih.gov/geo/series/")
public void testGSE155695()
                   throws IOException,
NoSingleCellDataFoundException

This dataset has technical replicates, this is not supported.

Throws:

IOException

NoSingleCellDataFoundException

testGSM4282408

@Test
@Disabled
public void testGSM4282408()
                    throws NoSingleCellDataFoundException,
IOException

This sample has duplicated cell IDs.

Throws:

NoSingleCellDataFoundException

IOException

testGSE109774

@Test
public void testGSE109774()
                   throws IOException

This series has numerous TAR archives with numerous small files within. We have a policy of skipping these if we skip enough entries within the archive.

Throws:

IOException

testGSE208742

@Test
@Tag("integration")
@NetworkAvailable(url="ftp://ftp.ncbi.nlm.nih.gov/geo/series/")
public void testGSE208742()
                   throws IOException,
NoSingleCellDataFoundException

This dataset has MEX data at both series and sample-levels. TODO: add support for this, see #1232

Throws:

IOException

NoSingleCellDataFoundException

testGSE244451

@Test
@Disabled
public void testGSE244451()
                   throws IOException,
NoSingleCellDataFoundException

This dataset as an array of ENUM, a raw.X and needs to be transposed.

Throws:

IOException

NoSingleCellDataFoundException

testGSE184506

@Test
@Tag("integration")
@NetworkAvailable(url="ftp://ftp.ncbi.nlm.nih.gov/geo/series/")
public void testGSE184506()
                   throws NoSingleCellDataFoundException,
IOException

This series has complete MEX files at the sample-level, but also provide MEX files in the series.

Throws:

NoSingleCellDataFoundException

IOException

testGSE125708

@Test
@Tag("integration")
@NetworkAvailable(url="ftp://ftp.ncbi.nlm.nih.gov/geo/series/")
public void testGSE125708()
                   throws NoSingleCellDataFoundException,
IOException

This series has samples lacking data.

Example: GSM3580744

Throws:

NoSingleCellDataFoundException

IOException

testGSE274772

@Test
@Tag("integration")
@NetworkAvailable(url="ftp://ftp.ncbi.nlm.nih.gov/geo/series/")
public void testGSE274772()
                   throws IOException

Throws:

IOException

testGSE165635

@Test
@Tag("integration")
@NetworkAvailable(url="ftp://ftp.ncbi.nlm.nih.gov/geo/series/")
public void testGSE165635()
                   throws IOException

The SRA metadata does not use the "TRANSCRIPTOMIC SINGLE CELL" library source, but keywords in the protocol indicate it is single-cell.

Throws:

IOException

testGSE169285

@Test
public void testGSE169285()
                   throws IOException

Throws:

IOException

testGSE256105

@Test
public void testGSE256105()
                   throws IOException

Throws:

IOException

testGSE251724

@Test
public void testGSE251724()
                   throws IOException

Throws:

IOException

testHasSingleCellDataInSra

@Test
@Tag("integration")
@NetworkAvailable(url="ftp://ftp.ncbi.nlm.nih.gov/geo/series/")
public void testHasSingleCellDataInSra()
                                throws IOException

Throws:

IOException

testHasSingleCellDataInCellXGene

@Test
@Tag("slow") @Tag("integration")
@NetworkAvailable(url="ftp://ftp.ncbi.nlm.nih.gov/geo/series/")
public void testHasSingleCellDataInCellXGene()
                                      throws IOException,
NoSingleCellDataFoundException

Throws:

IOException

NoSingleCellDataFoundException

testDownloadSingleCellDataInCellXGene

@Test
@Tag("integration")
@NetworkAvailable(url="ftp://ftp.ncbi.nlm.nih.gov/geo/series/")
public void testDownloadSingleCellDataInCellXGene()
                                           throws IOException,
NoSingleCellDataFoundException

Throws:

IOException

NoSingleCellDataFoundException

testDownloadSingleCellDataInCellXGeneWithoutACollectionId

@Test
@Tag("slow") @Tag("integration")
@NetworkAvailable(url="ftp://ftp.ncbi.nlm.nih.gov/geo/series/")
public void testDownloadSingleCellDataInCellXGeneWithoutACollectionId()
                                                               throws IOException,
NoSingleCellDataFoundException

Throws:

IOException

NoSingleCellDataFoundException

testGSE71585

@Test
@Tag("integration")
@NetworkAvailable(url="ftp://ftp.ncbi.nlm.nih.gov/geo/series/")
public void testGSE71585()
                  throws IOException

Throws:

IOException

testIsSingleNuclei

@Test
public void testIsSingleNuclei()
                        throws IOException

Throws:

IOException