tally-reads

This mode is used to tally the number of times sequences appear in a set of read files. Exact sequence comparison is performed. It is implemented by edu.cornell.med.icb.goby.modes.TallyReadsMode.java.

Mode Parameters

The following options are available in this mode

Flag	Arguments	Required	Description
`(-i\|--input)`	input	yes	Name of the compact input file.
`(-o\|--output)`	output	yes	Base name of the filter file output (the resulting file will be the output basename given plus "-keep.filter").
`--color-space`	n/a	no	When this switch is provided, the sequence are expected in color-space.

Comments

Leave a comment Trackback

Written by Adrian about 10 months ago.
Hi all. Many thanks for the Goby system – I only recently discovered the packages but they seems ideal for some QA manipulation on NGS data.
At this point I’m wondering whether version 1.9.7.2 is able to support the read volumes generated from illumina HiSeq data? As a test I tried to remove the degenerate reads from a lane of HiSeq MP data. This seemed to fail – I’m guessing because the number of reads perhaps exceeded the maximum?
java -Xmx3g -jar goby.jar -m tally-reads -i ~/data/MP-data/July5th_10Kb/SI/94_84_64_paired_trimmed_s_4_1_S_Irio_sequence.compact-reads -o myfilter
INFO GobyDriver – edu.cornell.med.icb.goby.modes.GobyDriver Implementation-Version: goby_1.9.7.2 (20110708173024)
INFO root – first pass: starting to collect inspect hash
INFO root – 3,877,888 items, 10s, 387,788.80 items/s; used/avail/free/total/max mem: 508.71M/2.35G/1.55G/2.06G/2.86G
INFO root – 7,880,704 items, 20s, 393,798.92 items/s; used/avail/free/total/max mem: 485.80M/2.38G/1.53G/2.01G/2.86G
INFO root – 11,919,360 items, 30s, 397,139.91 items/s; used/avail/free/total/max mem: 480.69M/2.38G/1.52G/2.00G/2.86G
INFO root – 15,889,408 items, 40s, 397,106.14 items/s; used/avail/free/total/max mem: 311.32M/2.55G/1.66G/1.97G/2.86G
INFO root – 19,994,624 items, 50s, 399,788.53 items/s; used/avail/free/total/max mem: 488.51M/2.37G/1.39G/1.88G/2.86G
INFO root – 24,112,128 items, 1m 0s, 401,781.75 items/s; used/avail/free/total/max mem: 350.64M/2.51G/1.49G/1.84G/2.86G
INFO root – 28,230,656 items, 1m 10s, 403,191.41 items/s; used/avail/free/total/max mem: 344.13M/2.52G/1.46G/1.81G/2.86G
INFO root – 32,350,208 items, 1m 20s, 404,231.07 items/s; used/avail/free/total/max mem: 466.56M/2.40G/1.31G/1.78G/2.86G
INFO root – 36,463,616 items, 1m 30s, 405,020.78 items/s; used/avail/free/total/max mem: 384.34M/2.48G/1.38G/1.77G/2.86G
INFO root – 40,570,880 items, 1m 40s, 405,583.07 items/s; used/avail/free/total/max mem: 161.38M/2.70G/1.57G/1.73G/2.86G
INFO root – 44,680,192 items, 1m 50s, 406,050.67 items/s; used/avail/free/total/max mem: 369.20M/2.49G/1.32G/1.69G/2.86G
INFO root – 48,790,528 items, 2m 0s, 406,442.09 items/s; used/avail/free/total/max mem: 207.11M/2.66G/1.46G/1.67G/2.86G
INFO root – 52,900,864 items, 2m 10s, 406,770.14 items/s; used/avail/free/total/max mem: 220.67M/2.64G/1.42G/1.64G/2.86G
INFO root – 57,007,104 items, 2m 20s, 407,045.32 items/s; used/avail/free/total/max mem: 187.99M/2.68G/1.44G/1.62G/2.86G
INFO root – 61,107,200 items, 2m 30s, 407,242.87 items/s; used/avail/free/total/max mem: 307.50M/2.56G/1.30G/1.60G/2.86G
INFO root – 64,987,136 items, 2m 40s, 406,040.17 items/s; used/avail/free/total/max mem: 464.49M/2.40G/1.13G/1.59G/2.86G
INFO root – 69,090,304 items, 2m 50s, 406,267.77 items/s; used/avail/free/total/max mem: 418.12M/2.45G/1.23G/1.64G/2.86G
INFO root – 73,210,880 items, 3m 0s, 406,571.26 items/s; used/avail/free/total/max mem: 404.90M/2.46G/1.22G/1.62G/2.86G
INFO root – 77,330,432 items, 3m 10s, 406,830.98 items/s; used/avail/free/total/max mem: 375.74M/2.49G/1.23G/1.60G/2.86G
INFO root – 81,440,768 items, 3m 20s, 407,041.02 items/s; used/avail/free/total/max mem: 508.91M/2.35G/1.08G/1.59G/2.86G
INFO root – 85,545,984 items, 3m 30s, 407,206.70 items/s; used/avail/free/total/max mem: 495.71M/2.37G/1.07G/1.57G/2.86G
INFO root – 89,650,176 items, 3m 40s, 407,343.42 items/s; used/avail/free/total/max mem: 404.64M/2.46G/1.15G/1.56G/2.86G
INFO root – 93,750,272 items, 3m 50s, 407,443.36 items/s; used/avail/free/total/max mem: 483.24M/2.38G/1.06G/1.54G/2.86G
INFO root – 97,860,608 items, 4m 0s, 407,570.82 items/s; used/avail/free/total/max mem: 424.83M/2.44G/1.11G/1.53G/2.86G
INFO root – 101,965,824 items, 4m 10s, 407,688.81 items/s; used/avail/free/total/max mem: 459.57M/2.40G/1.06G/1.52G/2.86G
INFO root – 106,060,800 items, 4m 20s, 407,755.21 items/s; used/avail/free/total/max mem: 425.15M/2.44G/1.09G/1.51G/2.86G
INFO root – 110,160,896 items, 4m 30s, 407,819.04 items/s; used/avail/free/total/max mem: 428.59M/2.43G/1.07G/1.50G/2.86G
INFO root – 114,247,680 items, 4m 40s, 407,849.72 items/s; used/avail/free/total/max mem: 406.76M/2.46G/1.09G/1.50G/2.86G
INFO root – 118,321,152 items, 4m 50s, 407,832.40 items/s; used/avail/free/total/max mem: 394.47M/2.47G/1.10G/1.49G/2.86G
INFO root – 122,392,576 items, 5m 0s, 407,809.41 items/s; used/avail/free/total/max mem: 395.80M/2.47G/1.09G/1.48G/2.86G
INFO root – 126,450,688 items, 5m 10s, 407,739.72 items/s; used/avail/free/total/max mem: 380.65M/2.48G/1.11G/1.49G/2.86G
INFO root – 130,528,256 items, 5m 20s, 407,740.25 items/s; used/avail/free/total/max mem: 421.87M/2.44G/1.06G/1.48G/2.86G
INFO root – 134,600,704 items, 5m 30s, 407,707.95 items/s; used/avail/free/total/max mem: 429.86M/2.43G/1.06G/1.49G/2.86G
INFO root – 138,672,128 items, 5m 40s, 407,691.33 items/s; used/avail/free/total/max mem: 381.65M/2.48G/1.10G/1.48G/2.86G
INFO root – 142,742,528 items, 5m 50s, 407,672.73 items/s; used/avail/free/total/max mem: 412.89M/2.45G/1.06G/1.48G/2.86G
INFO root – 146,820,096 items, 6m 0s, 407,668.27 items/s; used/avail/free/total/max mem: 397.90M/2.47G/1.09G/1.49G/2.86G
INFO root – 150,896,640 items, 6m 10s, 407,667.89 items/s; used/avail/free/total/max mem: 424.58M/2.44G/1.06G/1.48G/2.86G
INFO root – 154,967,040 items, 6m 20s, 407,651.38 items/s; used/avail/free/total/max mem: 398.96M/2.46G/1.10G/1.50G/2.86G
INFO root – 159,050,752 items, 6m 30s, 407,655.20 items/s; used/avail/free/total/max mem: 400.15M/2.46G/1.09G/1.49G/2.86G
INFO root – 163,120,128 items, 6m 40s, 407,635.23 items/s; used/avail/free/total/max mem: 401.22M/2.46G/1.08G/1.49G/2.86G
INFO root – 167,190,528 items, 6m 50s, 407,608.79 items/s; used/avail/free/total/max mem: 385.16M/2.48G/1.10G/1.48G/2.86G
INFO root – first pass finished.
Found 9939608 redundant hashCodes.
INFO root – second pass: actual read redundancy evaluation.
java.lang.ArrayIndexOutOfBoundsException: 82
at com.google.protobuf.ByteString.byteAt(ByteString.java:60)
at edu.cornell.med.icb.goby.reads.SequenceDigests.crc32(SequenceDigests.java:180)
at edu.cornell.med.icb.goby.reads.SequenceDigests.digest(SequenceDigests.java:96)
at edu.cornell.med.icb.goby.modes.TallyReadsMode.execute(TallyReadsMode.java:187)
at edu.cornell.med.icb.goby.modes.GenericToolsDriver.execute(GenericToolsDriver.java:171)
at edu.cornell.med.icb.goby.modes.GobyDriver.main(GobyDriver.java:54)
Written by Fabien Campagne about 10 months ago.
Hello Adrian,
You don’t seem to be running out of memory according to the log messages at the end of the first pass (385Mb used with 2.88Gb max memory). We also have run around this number of reads successfully with tally-reads, so the error seems to be related to your input file. The exception suggests that the read-length recorded in the read is longer than the actual length of sequence stored. Could you indicate how you created the compact-reads file? Also, try running compact-file-stats on the compact file to see if read lengths are constant. Perhaps you have trimmed the reads resulting in variable read length? Inspecting the code points to a possible bug when read lengths vary in a file. There is an easy fix if you have the source code, try adding the following statements
readLength = readEntry.getReadLength();
at line 176 of TallyReadsMode.java and rebuild the jar file.
Written by Adrian about 10 months ago.
Yes, indeed the reads were of variable length having been pre-trimmed for quality. I believe the fix you suggest above works well. Many thanks.

This site is protected by Comment SPAM Wiper.

Page optimized by WP Minify WordPress Plugin

Campagne Laboratory

tally-reads

Mode Parameters

Comments

Leave a Comment