🚀 go-pugleaf

RetroBBS NetNews Server

Inspired by RockSolid Light RIP Retro Guy

18 total messages Started by Leonard Erickson Wed, 06 Jan 1988 05:36
how do you tell encrytped data from random data?
#3933
Author: Leonard Erickson
Date: Wed, 06 Jan 1988 05:36
33 lines
2059 bytes
An interesting question has crossed my mind. If someone presents you with
an allegedly encrypted message, How can you tell if it really is encrypted
as opposed to being a bunch of random characters?

I know that transposition and *simple* substitution can be detected by
letter frequency analysis. But is a "flat" distibution evidence of random
data?

For my purposes, both "one-time pad" ciphers and anything that operates on
units other than characters can be considered random! If it is that complex,
then I'm not likely to crack it!

This is inspired by my fiinding some notes I made a year or so ago when I
and some friends were exchanging encrypted msgs over a public BBS. A few
of our friends cracked or first efforts and a typical "arms race" of better
ciphers followed by better analytic techniques ensued.

We finally came (after about 5 changes) came up with a scheme that only
one outsiders could read part of. Some of the other users started using
their own ciphers. The sysop didn't mind until someone started posting
HUGE messages which he suspected were just garbage. But the challenge
was, were they garbage or real messages?

Under his rules, messages (even encrypted!) were ok, junk wasn't. I made
a few attempts to analyze them, but while I was fairly certain, I could
never be sure. (when an 8k msg uses the 26 letters so evenly that the
spread better most used and least used is 12, you get *real* suspicious :-)

-- 
Leonard Erickson		...!tektronix!reed!percival!bucket!leonard
CIS: [70465,203]
"I used to be a hacker. Now I'm a 'microcomputer specialist'.
You know... I'd rather be a hacker."
Re: how do you tell encrytped data from random data?
#3957
Author: Doug Gwyn
Date: Fri, 08 Jan 1988 06:37
29 lines
1864 bytes
In article <660@bucket.UUCP> leonard@bucket.UUCP (Leonard Erickson) writes:
>An interesting question has crossed my mind. If someone presents you with
>an allegedly encrypted message, How can you tell if it really is encrypted
>as opposed to being a bunch of random characters?

The only foolproof way is to decrypt it.

>I know that transposition and *simple* substitution can be detected by
>letter frequency analysis. But is a "flat" distibution evidence of random
>data?

No, most really good encryption systems pretty much randomizes the cipher
stream, insofar as simple statistical tests can determine.  Conversely,
just because some putative cipher stream appears nonrandom does not mean
that it has meaningful underlying plaintext.  It could be a hoax.  There
are those who think the two undeciphered Beale cryptograms are hoaxes.

>For my purposes, both "one-time pad" ciphers and anything that operates on
>units other than characters can be considered random! If it is that complex,
>then I'm not likely to crack it!

You can't crack a true one-time pad system by analysis anyway, no matter
how competent a cryptanalyst you are.

>(when an 8k msg uses the 26 letters so evenly that the
>spread better most used and least used is 12, you get *real* suspicious :-)

Flatter-than-random distributions are not necessarily hoaxes.
They're actually pretty easy to obtain; Mil Cryp has an example.
Re: how do you tell encrytped data from
#3935
Author: leather
Date: Fri, 08 Jan 1988 19:15
44 lines
3064 bytes
> /* Written 11:36 pm  Jan  5, 1988 by leonard@bucket.UUCP ...   */
> /* ---------- "how do you tell encrytped data from" ---------- */
> An interesting question has crossed my mind. If someone presents you with
> an allegedly encrypted message, How can you tell if it really is encrypted
> as opposed to being a bunch of random characters?
> 
> [...] 
> 
	Although the _suspicious_ nature of an even distribution of letters
may be a give-away indication of *JUNK* messages for amateur and "semi-pro"
encryption schemes, it cannot be absolute proof.

	I have dabbled some in the crypto-analysis field, myself.  It is not
difficult to think of some methods for creating *SMOOTH* distributin messages:
	1.  massage the message with your existing encryption scheme,
	2.  perform a distribution analysis on the encoded result,
	3.  obtain the complementary distribution analysis - for *SMOOTHNESS*,
	4.  apply a padding of additional characters to the previously
 	    encoded result, yielding the smoothed encrypted message:
		a.  padding could be in the form of "encasing" the previously
		    encoded result (either/both ends of message by line, block,
		    ...),
		b.  or "enbedding" the characters in some simple/complex 
                    pattern.

	The real question is why bother, except to annoy or distract?
If you have a sufficiently effetive cypher to begin with, why add complexity
to its encoding/decoding process?  If you are going to hide a *REAL* message
in a diversionary barrage of *JUNK* messages, you could just as well hide it
in *JUNK* that used ordinary English, French, Russian, Chinese, ... sentence
packages, in which case the _normal_ distribution patterns are going to lead
to equally wrong conclusions.  The point is, there are ever so many ways for
going about this business of encryption that one analysis technique alone is
insufficient to determine what *IS* or *IS_NOT* encrypted data.  Even if one
goes to the trouble of using a less complex technique for encryption of some
message, who is to say that the _message_ was not the *JUNK*.  GIGO (garbage
in, garbage out)?  It may be *JUNK*, but it _was_ encrypted!  But then, ....

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  David A. Leatherman                  ~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  (ARPA: leather@osiris.cso.uiuc.edu)  ~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~         leathrmn@uxe.cso.uiuc.edu     ~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Re: how do you tell encrytped data from random data?
#3938
Author: Marcus J. Ranum
Date: Sat, 09 Jan 1988 19:55
20 lines
1499 bytes
	Reminds me of a wonderful concept in a book by Stanitslaw Lem
(Memoirs Found in a Bathtub):

	The underground of the Pentagon is all that remains after the end of
the world, but the spooks are sure they have been infiltrated. One of the best
tools they have is a super-dooper computer than can decrypt *ANYTHING* if it
tries hard enough. In fact, it can take plain text, and find the secret and
subversive messages hidden in it - even if there weren't any to begin with...

	Another problem with Gywn's suggestion that the only way to tell if it
is encrypted (decrypt it) - how can you tell that it is really "meaningful" ?
Anyone who has listened to first year latin students trying to "decrypt"
latin will realize that garbage in does not always imply the same garbage out.

--mjr();
-- 
------------------------------------------------------------------------------
...ich bin in einem dusenjet ins jahr 53 vor chr...ich lande im antiken Rom...
                     einige gladiatoren spielen scrabble...ich rieche PIZZA...
Re: how do you tell encrytped data from random data?
#3939
Author: Henry Schaffer
Date: Sun, 10 Jan 1988 02:22
14 lines
1100 bytes
These days it is quite common to compress text (and other files too.)
The various compressions schemes such as Huffman encoding and Lempel-
Ziv-Welch tend to even out (smooth out) the character distribution
(i.e., they hope to maximize the information transmitted per character,
or per byte).  If this file is then encrypted, the smoothness may
end up in the encrypted data.  

Conversly, random data need not have a smooth character distribution.
(It is common to confuse "random" with "equi-probable".  However, even
though most examples given in courses of "random" are "equi-probable
there is no necessary connection.)

--henry schaffer  n c state univ
Re: how do you tell encrytped data from
#3944
Author: gordon
Date: Sun, 10 Jan 1988 07:24
75 lines
4817 bytes
> An interesting question has crossed my mind. If someone presents you with
> an allegedly encrypted message, How can you tell if it really is encrypted
> as opposed to being a bunch of random characters?

You can't.  And if I wanted to really screw things up, and had plenty of
transmission bandwidth, I could use an encoding method that was mostly,
but not completely noise.  For example, each netnews article I post could 
have encoded in it one bit, which is either 1 or 0 depending on whether the
number of lines in it plus the number of newsgroups it is cross-posted to
is odd or even.  I have to post about 200 netnews articles to transmit one 
sentence.  I defy anyone to prove that there is really a message!
For a less blatant waste of transmission bandwidth, how about using the
"parity" bits only to contain the message, and the lower 7 bits contain
random quotes from "talk.abortion" and "talk.bizarre"?

> I know that transposition and *simple* substitution can be detected by
> letter frequency analysis. But is a "flat" distibution evidence of random
> data?

No.  Among other things, crypt(1) output gives a fairly flat distribution
in its 256-character output alphabet.  (the version of crypt I am thinking of 
is based on a WW-II rotor machine, and uses DES only to scramble the user's 
typed key.  (You don't need to crack the DES to crack the encryption).  This 
version was in System III, among others).  In fact, it gives an absolutely 
flat distribution for any message consisting of one character repeated a 
multiple of 256 times and any key.  (If not a multiple of 256, the 
distribution is as flat as you can get with that message length).

You also get a flat distribution, more or less, by encoding techniques 
as "encode the i th character of the message by rotating it i positions in 
the alphabet", assuming the message was reasonably long.  This doesn't even 
have an encryption key.  A "real" encryption might use a keyword J characters
long, and encode the i th character of the message by rotating it by its
position in the message (i) plus an amount determined from the (i mod J)th 
character of the key.  For messages that are long relative to the key length 
and the alphabet size, this will give a fairly flat distribution.  You will
be able to detect distribution patterns by taking every (J * alphabet size)th
character, and looking at the frequency distribution for that.  

> For my purposes, both "one-time pad" ciphers and anything that operates on
> units other than characters can be considered random! If it is that complex,
> then I'm not likely to crack it!

I'm not quite sure how to interpret this.  Would this mean that running the
message through uuencode(1) (which maps every 3 characters of 8 bits each
into 4 characters out of a 6-bit 64-character alphabet) is too complex?  It 
doesn't even have an encryption key!  Running stuff through uuencode does tend 
to flatten out the distribution somewhat, but a message consisting mostly of
the letter x will still have several high-frequency characters.

To REALLY flatten out the distribution, there are several popular UNIX and
DOS utilities to do this, under the classification "data compression programs".
These include pack, compress, ARC, and compact.  If you don't like the
use of all 8 bits in the output, run the result through a filter to transform
it to a smaller alphabet.  These include things like uuencode/uudecode, 
btoa/atob (distributed with compress 4.0), encode/decode (distributed with B 
news 2.11), and, if you're really desparate, od, the octal dump program.
(A few years back, someone posted a program to take an octal dump and 
re-create the file it was a dump of).  

These programs do not encrypt (that is, there is no encryption key, and just
knowing which programs are used and in what order is sufficient to recover
the plaintext.  To get reasonable security, transform the text in several
stages:

plaintext -> compress -> your favorite encryption -> uuencode -> ciphertext

Compress first, then encrypt.  You save on message transmission costs and
you mess up some clues used to attack the encryption - such as "all characters
going into the encryption probably have the high-order bit turned off", because
the compression removed much of the redundancy.

					Gordon Burditt
					...!ihnp4!sys1!sneaky!gordon
Re: how do you tell encrytped data from random data?
#3941
Author: Doug Gwyn
Date: Mon, 11 Jan 1988 09:50
44 lines
3137 bytes
In article <1499@osiris.UUCP> mjr@osiris.UUCP (Marcus J. Ranum) writes:
>Another problem with Gywn's suggestion that the only way to tell if it
>is encrypted (decrypt it) - how can you tell that it is really "meaningful"?

It's not that hard in practice.  Witness the following, which have been
uniquely cryptanalyzed into plaintext which all involved agreed was
correct (and in cases where it could be checked with the encryptor,
its correctness was verified):
	(a)  the Chaucer transposition with spelling variations that
	     was posted to this newsgroup not long ago;
	(b)  "challenger" cryptograms
Some of the latter are extremely short, and have plaintext that nobody
in his right mind would really use.  I've personally broken quite short
messages, some 25 letters long, encrypted by some of the simpler
methods.  Usually, student cryppies would all get essentially the same
result (sometimes off in one or two letters) for cryptanalysis of
practice cryptograms, if they were able to crack them at all.

It is extremely hard to make up a cryptogram of modest length that
has two valid but different decryptions.  In fact, I don't know of
any of length more than a couple of dozen letters, although I can
imagine some ways to try to do this for somewhat longer messages.
This would make an interesting puzzle for someone.

On the other hand, there are people such as those who find Baconian
ciphers in Shakespeare's works, who put the message there themselves.
It's pretty easy to tell when this happens; either the decipherment
method is ambiguous, so the message could never have been
realistically expected to be read even by its intended recipient,
or so many "corrections" and otherwise unjustified special twists
on the decipherment scheme have been introduced that the
information content of the decipherment rivals that of the supposed
message being extracted.  It is interesting to note that William
F. Friedman, perhaps the greatest cryptanalyst of all time, and his
future wife got their start in the field by being employed at "Col."
Fabyan's Riverbank Labs, which was trying to find just such hidden
messages in Shakespeare.  The Friedmans showed how the same
methodology could be used to produce messages with precisely
opposite meaning, i.e. the methodology was making up the whole
content of the messages based on the analyst's preconceptions
about what was to be found.  On the other side of this issue,
several rum-runner codes (generally much more ambiguous to crack
than ciphers) were broken by the Friedmans, and they were able to
demonstrate their methodology satisfactorily to the federal courts.
Re: how do you tell encrypted data from random data?
#3943
Author: Jim Gillogly
Date: Mon, 11 Jan 1988 17:25
36 lines
2503 bytes
In article <1499@osiris.UUCP> mjr@osiris.UUCP (Marcus J. Ranum) writes:
>	Another problem with Gywn's suggestion that the only way to tell if it
>is encrypted (decrypt it) - how can you tell that it is really "meaningful" ?
>Anyone who has listened to first year latin students trying to "decrypt"
>latin will realize that garbage in does not always imply the same garbage out.

With the kinds of ciphers Doug Gwyn was referring to, you can tell.  There
can be ambiguity if the message is short enough, but the usual case is that
you figure out the system and the keyword, use it to break part of the
message, and the rest of the message falls out nicely.  For example, we
could apply Mr.  Ranum's argument to DES with a key search: we've tried
our 7th New Collegiate on a 64-bit block, and discover that the key
"methionine" (suitably folded) results in legal 7-bit ascii plaintext
looking like "he purpo" for that 64-bit block.  Proceeding further, we
find that "rhinencephalon" (also folded) results in "73ux&aUc", which is
also legal ascii and could easily be part of a uuencoded piece of source
code.  In order to see whether either of these is the correct keyword, we
try them on the next 64-bit block...

There can be ambiguity in decrypting some kinds of cipher, such as a "book
code".  A lot of people have "decrypted" the Beale Cyphers without being
able to prove it -- and unless they come up with a key text that supports
their own view (or they dig up the treasure, of course) nobody else will
believe them.

For interesting reading on times when people have had to prove (i.e.
demonstrate to a court) that they got the right reading, try "The
Codebreakers" (David Kahn), "The Zimmermann Telegram" (Barbara Tuchman),
and "The Shakespearean Ciphers Examined" (William and Elizebeth Friedman).
[Elizebeth Friedman specialized in helping convict drug runners by
breaking their ciphers; she would testify in court about the validity of
her decryption.]
-- 
	Jim Gillogly
	{hplabs, ihnp4}!sdcrdcf!randvax!jim
	jim@rand-unix.arpa    [HASA: U (Spam) division]
Re: how do you tell encrytped data from random data?
#3960
Author: Phil R. Karn
Date: Mon, 11 Jan 1988 23:30
25 lines
1895 bytes
Re random and encrypted messages: There is a strong connection between
randomness and cryptography. "Randomness" is in the eye of the beholder.
All it means for you to say that something is "random" is that *you*
can't predict what the next element in the sequence is, not that it
doesn't contain some underlying redundancy that someone else with
additional information (or computing power) might discover.  It
therefore follows that any strong encryption algorithm would necessarily
produce what appeared to be totally random ciphertext. Feed a counter
into DES and you have an excellent random number generator;
unfortunately it is usually too slow to be practical.

> On the other hand, there are people such as those who find Baconian
> ciphers in Shakespeare's works, who put the message there themselves.
> It's pretty easy to tell when this happens; either the decipherment
> method is ambiguous, so the message could never have been
> realistically expected to be read even by its intended recipient,
> or so many "corrections" and otherwise unjustified special twists
> on the decipherment scheme have been introduced that the
> information content of the decipherment rivals that of the supposed
> message being extracted.

Quite true. I know I'm going to get flamed for this, but this is *precisely*
what people do all the time with the Bible.

Phil
Re: how do you tell encrytped data from
#3946
Author: truett
Date: Tue, 12 Jan 1988 10:45
29 lines
2049 bytes
Randomness or flatness of the probability distribution of the result is neither
necessary nor sufficient for a good encryptor.

First, it is not necessary.  Consider the one time pad.  In a binary stream, a
one time pad will convert "random" streams to "nonrandom" ones just as often as
it converts "nonrandom" (presumably plaintext) streams to apparently "random"
ones.  The only way to camouflage this effect is to increase the size of the
apparent alphabet or state space of the message channel (somebody correct me if
I'm wrong here.).

It isn't sufficient either.  Consider the linear congruential pseudorandum
number generator:  y = (ax + b) mod (n).  For suitable a,b,n, you can use the
successive characters of the message as values of x and the resulting stream of
y values will be quite random.  Another way is to use a data compression scheme
like the ARC system used on many electronic bulletin board systems.  The
reduction of redundancy in a data stream makes it look more like noise than the
original.

Actually, good random number generators make quite good encryptors.  So do
ECC generators.  Remember that Shannon demonstrated that the closer a noisy
channel got to zero error, the more the data stream has to look like noise.
Ergo:  apply a really good ECC to the data stream (thus adding redundant data,
see my comment above) and "BINGO" you have a good encryptor.

Again, the answer is NO, you can't tell noise and encrypted data apart without
a known encryption system.  In the latter case, you tell by decrypting.

Truett Lee Smith, Sunnyvale, CA
UUCP:  truett@cup.portal.com
Re: how do you tell encrytped data from
#3948
Author: Leonard Erickson
Date: Fri, 15 Jan 1988 05:07
72 lines
4745 bytes
In article <-63293657@sneaky> gordon@sneaky.UUCP writes:
<> I know that transposition and *simple* substitution can be detected by
<> letter frequency analysis. But is a "flat" distibution evidence of random
<> data?
<
<No.  Among other things, crypt(1) output gives a fairly flat distribution
<in its 256-character output alphabet.  (the version of crypt I am thinking of 
<is based on a WW-II rotor machine, and uses DES only to scramble the user's 
<typed key.  (You don't need to crack the DES to crack the encryption).  This 
<version was in System III, among others).  In fact, it gives an absolutely 
<flat distribution for any message consisting of one character repeated a 
<multiple of 256 times and any key.  (If not a multiple of 256, the 
<distribution is as flat as you can get with that message length).
<
<You also get a flat distribution, more or less, by encoding techniques 
<as "encode the i th character of the message by rotating it i positions in 
<the alphabet", assuming the message was reasonably long.  This doesn't even 
<have an encryption key.  A "real" encryption might use a keyword J characters
<long, and encode the i th character of the message by rotating it by its
<position in the message (i) plus an amount determined from the (i mod J)th 
<character of the key.  For messages that are long relative to the key length 
<and the alphabet size, this will give a fairly flat distribution.  You will
<be able to detect distribution patterns by taking every (J * alphabet size)th
<character, and looking at the frequency distribution for that.  
 
All the above examples, including the crypt that simulates a rotor machine
are polyalphabetic substitutions. I have program that will do analysis of
a text by calculating the variance for all the key lengths up to a user 
specified limit. It outputs the length of with the "best fit". 

On both random text and the suspect text, it always output the limit length
as the best fit. And the fit was none too good.

<I'm not quite sure how to interpret this.  Would this mean that running the
<message through uuencode(1) (which maps every 3 characters of 8 bits each
<into 4 characters out of a 6-bit 64-character alphabet) is too complex?  It 
<doesn't even have an encryption key!  Running stuff through uuencode does tend 
<to flatten out the distribution somewhat, but a message consisting mostly of
<the letter x will still have several high-frequency characters.
<
<To REALLY flatten out the distribution, there are several popular UNIX and
<DOS utilities to do this, under the classification "data compression programs".
<These include pack, compress, ARC, and compact.  If you don't like the
<use of all 8 bits in the output, run the result through a filter to transform
<it to a smaller alphabet.  These include things like uuencode/uudecode, 
<btoa/atob (distributed with compress 4.0), encode/decode (distributed with B 
<news 2.11), and, if you're really desparate, od, the octal dump program.
<(A few years back, someone posted a program to take an octal dump and 
<re-create the file it was a dump of).  

Well, I've never even seen any _hints_ on how to attack a cipher where the 
plaintext characters can't be "localized". Since you can't do statistical
analysis on the characters (at least not and get results that _mean_ 
anything!! :-) it would appear that you are reduced to trial and error.

Uuencode and atob are at the extreme edge of the ability range expected of
the probable originators of those old messages. Compress and ARC were 
out of the question! (we're talking c-64s and BASIC here!).  

A favorite that has a fair amount of popularity, is XORing the plaintext
with a key on a byte-by-byte basis. It is simple, and you decode by running
the ciphertext thru the encryptor again. Of course it is only as secure as
key.

You would also be surprised at how long you can stump even a skilled amateur
with something silly like switching character sets. (eg using EBCDIC where
ASCII is expected)      
-- 
Leonard Erickson		...!tektronix!reed!percival!bucket!leonard
CIS: [70465,203]
"I used to be a hacker. Now I'm a 'microcomputer specialist'.
You know... I'd rather be a hacker."
Re: how do you tell encrytped data from
#3962
Author: Doug Gwyn
Date: Fri, 15 Jan 1988 15:10
19 lines
1543 bytes
In article <2415@cup.portal.com> truett@cup.portal.com writes:
>In a binary stream, a one time pad will convert "random" streams to "nonrandom"
>ones just as often as it converts "nonrandom" (presumably plaintext) streams to
>apparently "random" ones.  The only way to camouflage this effect ...

This is quite misleading.  There is no effect to "camouflage", since
the so-called effect being described is the purely theoretical one
where the entire space of possible messages is considered.  Useful
messages are a tiny fraction of this space.  In practice, the cipher
stream does not really exhibit "nonrandomness" (on the average) when
the one-time pad is applied to random plaintext.

>Actually, good random number generators make quite good encryptors.

Actually, assuming you're still talking about such pseudo-random
sequences as are produced by linear congruential feedback shift
registers or other such simple schemes, they make TERRIBLE encryptors.
They have far too much structure with far too small a key.  Therefore
they're quite easy to break, using the proper tools.
Re: how do you tell encrytped data from
#3966
Author: truett
Date: Mon, 18 Jan 1988 09:14
17 lines
1385 bytes
The idea that the set of possible plaintext messages is small relative to the
set of possible messages is not always valid.  How about the encryption of
binary object codes or the vectorized files of the plans of a classified
mechanical device?  A good cryptographer must assume that any possible message
has a chance of eventually appearing for encryption.

So long as the encryption process is one-to-one between the spaces of possible
plaintext messages and cyphertext messages, then "nonrandom" (whatever that
means!) messages are encrypted into "rnadom" messages just as often as the
"random" ones are encrypted into "nonrandom" ones.  What *is* rare, so long as
the nonrandom messages are a smaller space than the space of all messages, is
the event in which a "nonrandom" message is encrypted into a "nonrandom"
message.  There are exceptions:  remember what the Germans thought of Navajo?

Truett Lee Smith, Sunnyvale, CA
UUCP:  truett@cup.portal.com
Re: how do you tell encrytped data from random data?
#3953
Author: Brian Chapman Mx321
Date: Mon, 18 Jan 1988 23:34
38 lines
2091 bytes
In article <660@bucket.UUCP> leonard@bucket.UUCP (Leonard Erickson) writes:
< An interesting question has crossed my mind. If someone presents you with
< an allegedly encrypted message, How can you tell if it really is encrypted
< as opposed to being a bunch of random characters?

It is the goal of any good encryption technique, in this class
of techniques, to make the data look random.
Any sort of pattern in the encrypted data is a crack you can
drive a decryption wedge into.

< I know that transposition and *simple* substitution can be detected by
< letter frequency analysis. But is a "flat" distibution evidence of random
< data?
< For my purposes, both "one-time pad" ciphers and anything that operates on
< units other than characters can be considered random! If it is that complex,
< then I'm not likely to crack it!

Yet UNIX crypt is a simple substitution code with the extra twists
that the offset in the file is added to the character (mod 256)
and you change substution tables every 256 characters.
This tends to flatten out a naively taken distrubution.

< (when an 8k msg uses the 26 letters so evenly that the spread
< better most used and least used is 12, you get *real* suspicious :-)
  ^^^^^^ you mean "between"?

You mean the character counts vary between 309 and 321.
Sounds about right to me for random looking data.

Artificialy flat distributions mean that the data is
either a fraud or the encryption method pads with extra
characters.

-- 
			   uunet!-\
Brian Chapman		microsof!-->sco!chapman
			   ihnp4!-/
Pay no attention to the man behind the curtain!  -- The Great & Powerfull Oz
Re: how do you tell encrytped data from random data?
#3955
Author: Henry Schaffer
Date: Wed, 20 Jan 1988 04:53
26 lines
1680 bytes
In article <275@sysco>, chapman@sco.COM (Brian Chapman Mx321) writes:
> In article <660@bucket.UUCP> leonard@bucket.UUCP (Leonard Erickson) writes:
> < An interesting question has crossed my mind. If someone presents you with
> < an allegedly encrypted message, How can you tell if it really is encrypted
> < as opposed to being a bunch of random characters?
> ... 
> You mean the character counts vary between 309 and 321.
> Sounds about right to me for random looking data.
  
I'm taking "random" as meaning multinomial with equi-probability.  In
this case - looking at any one character should be found about 8k/26=307
times, with a standard deviation of about 17.  That makes a spread of 12
between the most and least frequent of the 26 look quite small, indeed.

> Artificialy flat distributions mean that the data is
> either a fraud or the encryption method pads with extra
> characters.

Hmm, is this correct?  I don't know how flat the output of something like
Huffman encoding can be.  I know it isn't encryption - but perhaps something
like this in conjunction with encryption would give a flatter-than-random
character distribution without character padding.
> -- 
> Brian Chapman		microsof!-->sco!chapman

--henry schaffer  n c state univ
Re: how do you tell encrytped data from random data?
#3967
Author: duncs home
Date: Wed, 20 Jan 1988 15:54
18 lines
1206 bytes
In article <660@bucket.UUCP> leonard@bucket.UUCP (Leonard Erickson) writes:
>                               ...
>I know that transposition and *simple* substitution can be detected by
>letter frequency analysis. But is a "flat" distibution evidence of random
>data?

No.  Consider this simple substitution cipher:

'a' = 'abcdefghijklmnopqrstuvwxyz'
'b' = 'bcdefghijklmnopqrstuvwxyza'
'c' = 'cdefghijklmnopqrstuvwxyzab'
             ...
'z' = 'zabcdefghijklmnopqrstuvwxy'

The frequency distribution of the letters will be absolutely flat, yet it
does contain a message.  This shows that a flat distribution does not mean
absence of a message (and also that flat distribution, by itself, does not
guarantee a good cipher!)
Re: how do you tell encrytped data from random data?
#3958
Author: Michael Mauldin
Date: Wed, 20 Jan 1988 22:52
22 lines
1586 bytes
In article <275@sysco>, chapman@sco.COM (Brian Chapman Mx321) writes:
> Artificialy flat distributions mean that the data is
> either a fraud or the encryption method pads with extra
> characters.

The encryption algorithm might very easily generate purposefully "flat"
distributions by a variety of methods.  You could switch tables after
every block to shuffle "overused" bytes to less commonly used meanings.
You can even guarantee perfectly flat counts by not altering the
"meaning" of each byte.  This turns out to require extra bytes, since
once you make a decision that "I've used \035 enough, now", the
remaining bytes encode fewer bits.  Still, the result would be neither
fraudulent nor padded with extra characters (the extra bytes would be
"necessary").

If you got a really good encryption algorithm, this would be a
pointless add-on feature, but that doesn't mean that seomone else isn't
doing it.

Michael L. Mauldin (Fuzzy)		Department of Computer Science
ARPA: Michael.Mauldin@NL.CS.CMU.EDU	Carnegie-Mellon University
Phone: (412) 268-3065			Pittsburgh, PA  15213-3890
Re: how do you tell encrytped data from random data?
#3970
Author: Jack Morrison
Date: Fri, 22 Jan 1988 16:16
44 lines
1955 bytes
In article <39394@sun.uucp> dunc@sun.UUCP (duncs home) writes:
>In article <660@bucket.UUCP> leonard@bucket.UUCP (Leonard Erickson) writes:
>>                               ...
>>I know that transposition and *simple* substitution can be detected by
>>letter frequency analysis. But is a "flat" distibution evidence of random
>>data?
>
>No.  Consider this simple substitution cipher:
>
>'a' = 'abcdefghijklmnopqrstuvwxyz'
>'b' = 'bcdefghijklmnopqrstuvwxyza'
>'c' = 'cdefghijklmnopqrstuvwxyzab'
>             ...
>'z' = 'zabcdefghijklmnopqrstuvwxy'
>
>The frequency distribution of the letters will be absolutely flat, yet it
>does contain a message.  This shows that a flat distribution does not mean
>absence of a message (and also that flat distribution, by itself, does not
>guarantee a good cipher!)


Cute, but keep in mind that a "distribution" can be more than just a histogram
of letter frequencies; the example cipher would have a very lopsided
distibution in other measurements, for example digram frequencies.
(I realize the proposal was not meant as a good cipher, and I'm just 
adding to the discussion; this is NOT a flame, I just don't see where to
stick the :-) !)

[What?
[The
[followup
[has
[to
[be
[longer
[than
[the
[original?
[Than
[eat
[this!]-- 
Jack C. Morrison	Jet Propulsion Laboratory
(818)354-1463		jackm@jpl-devvax.jpl.nasa.gov
"The paycheck is part government property, but the opinions are all mine."
Thread Navigation

This is a paginated view of messages in the thread with full content displayed inline.

Messages are displayed in chronological order, with the original post highlighted in green.

Use pagination controls to navigate through all messages in large threads.

Back to All Threads