🚀 go-pugleaf

RetroBBS NetNews Server

Inspired by RockSolid Light RIP Retro Guy

Article View: demon.tech.mac
Article #1136

Re: Removing duplicates within text files?

#1136
From: Alan W. Frame
Date: Mon, 11 May 1998 00:00
29 lines
1966 bytes
Rebecca and Rowland <real-address-in-sig@rhu.barb.foobar> wrote:

> Paul Womar <{$PW$}@pwomar.demon.co.uk> wrote:
> 
> > I have found that when I get the list of New group from the news server
> > that many appear for several sessions and so are added to my newsgroups
> > list many times over thus making it quite a bit longer than it should
> > be, is there any program that will easily strip out the duplicates, it's
> > a normal text file, alphabetically sorted with one entry per, any ideas?
> 
> Off the top of my head, I can't think of an *easy* solution.  You could
> do the job with A.N.Other programming language; Applescript and the
> Scriptable Text Editor; tcl within the Alpha text editor; or Mac Perl
> spring to mind as obvious possibilities.

Even easier - fire up Excel[1] and do an "Extract Distinct" (IIRC, on
v4[2]) or a "Data Filter:Advanced filter:Unique records only"(v5) on the
file.

The real Q&D approach is just to open it in Excel, sort and eyeball
search for duplicates and delete those rows.

rgds, Alan
[1] Yes, even I use Excel, but it's almost always for this sort of text
manipulation 'cos I'm too lazy to write some Perl and haven't used MPW
Shell for years....
[2] 'tho if it's a list of all groups, ISTR v4 Barfs at >16K rows..
-- 
Technical Details
Message-ID: <1998051119431528145@newtechd.demon.co.uk>
Path: arc1-ca.pugleaf.net!unknown!not-for-mail
References: <1d8qhru.1tlty0r18utg0vN@pwomar.demon.co.uk> <1d8qwhh.1mgakcv1r35nbqN@p25.nas1.is2.u-net.net>

Show Full Headers
Complete Article Headers
X-Google-Language: ENGLISH,ASCII-7-bit
X-Google-Thread: 10b5a2,5753a18ed13d861d
X-Google-Attributes: gid10b5a2,public
From: alan.frame@acm.org (Alan W. Frame)
Subject: Re: Removing duplicates within text files?
Date: 1998/05/11
Message-ID: <1998051119431528145@newtechd.demon.co.uk>
X-Deja-AN: 352292671
References: <1d8qhru.1tlty0r18utg0vN@pwomar.demon.co.uk> <1d8qwhh.1mgakcv1r35nbqN@p25.nas1.is2.u-net.net>
X-NNTP-Posting-Host: newtechd.demon.co.uk:158.152.58.83
X-Complaints-To: abuse@demon.net
X-Trace: news.demon.co.uk 894912253 nnrp-11:8321 NO-IDENT newtechd.demon.co.uk:158.152.58.83
Organization: Coder Of Fortune
Newsgroups: demon.tech.mac