Thursday, April 25, 2013

A Case Of Mysterious NETBIOS Traffic from Domain Controller

Few days back our Security team started reporting NETBIOS traffic going out to Internet from Domain Controllers in one of our sites... Weird huh?? Were the servers compromised??? Oh, that wont be good for a Domain Controller .. :(

Anyways, we all started looking into Tipping Point logs provided by the security team to understand the reason why our DCs were sending out NBT packets outside our network. We saw good number of packets flowing out from our DCs to some unknown public subnets, which wasn't good for a server which we thought or knew doesn't and shouldn't have exposure to External world.

Anyways, we planned to gather data for few days and investigate further... We picked up one of the Domain Controllers and started investigating...Awesome, our logic did gave us a big hint.. We found out the server were sending packets thrice a day.. quite consistent of time.. 5AM, 6PM  & 11PM.. and it was doing the same everyday.. So we started looking for any suspicious process running on the box or any specific activity its doing that could be generating that traffic but unfortunately we found nothing..  We went through Eventlogs as per the trend we found but didnt see anything unusual.. and nothing we could find in the Task Scheduler that could trigger something ..

Hmm, now we were left to run Network Monitor on the Domain Controller and see what process is sending those packets.. So, we downloaded Network Monitor and captured data .. As you know, running Netmon on a production server isn't good option as it can quickly consume server resources and may bring down the server. So, we decided to filter the traffic and keep a close eye on Netmon ;-)

Here;s the filter we used(SSS.SSS.SSS.SSS is the Source IP address classs and DDD.DDD.DDD.DDD is the destination IP address class)

((ipv4.SourceAddress & ==SSS.SSS.SSS.SSS)


((ipv4.DestinationAddress & == DDD.DDD.DDD.DDD)

Great, now we started looking thru Network packet to find the trace and we found SYSTEM process sending out that traffic..

Network Packet Frame
  Frame: Number = 11, Captured Frame Length = 92, MediaType = ETHERNET

- Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[DD-DD-DD-DD-DD-DD],SourceAddress:[SS-SS-SS-SS-SS-SS]
  + DestinationAddress: [DD-DD-DD-DD-DD-DD]
  + SourceAddress: [SS-SS-SS-SS-SS-SS]
    EthernetType: Internet IP (IPv4), 2048(0x800)
+ Ipv4: Src = SS..SS.SS.SS, Dest = DD.DD.DD.DD, Next Protocol = UDP, Packet ID = 20651, Total IP Length = 78
+ Udp: SrcPort = NETBIOS Name Service(137), DstPort = NETBIOS Name Service(137), Length = 58
- Nbtns: Query Request for *<00><00><00><00><00><00><00><00><00><00><00><00><00><00><00> <0x00>Workstation Service
    TransactionId: 49998 (0xC34E)
  + Flag: 0 (0x0)
    QuestionCount: 1 (0x1)
    AnswerCount: 0 (0x0)
    NameServiceCount: 0 (0x0)
    AdditionalCount: 0 (0x0)
  + NbtNsQuestionSectionData: 

Now, we needed to find out what's running under System or what is System process doing. The only way for us to find it out was run Process Monitor and capture more logs. We knew the trend of the activity and we made sure we run Network Monitor and Process Monitor well before the server starts sending out the packets again..

My colleague enabled the logging and save the logs for me to review as he thinks UK shift is on a much more leisurely schedule.. :)

So, i pulled up the logs from the server and started going thru it.. it was like finding a needle in the haze.. lol
I started Process Monitor and opened the saved log file.. and just to let you all know that by default Process Monitor has a filter to hide SYSTEM process activity.

So, please make sure you remove that filter as i spent first 10 mins looking for system process in there.. :)

Then i searched one of the destination IPs and found few UDP requests sent by the system process and just above it i found UDP receive request made to DNS, interesting.. huh!!

Now, i had to go back and look in the Network Monitor logs and find out the packets.. So, i started searching under system process and found the Network Packet sent out.. then i looked under DNS and found a machine sending a request to DNS server to resolve one of the suspicious  subnet IP addresses.

Here's the Packet frame showing a DNS request from a machine

SS.SS.SS.SS = Source Machine IP address
DC.DC.DC.DC = Domain Controller IP address
SIP.SIP.SIP.SIP = Suspicious UP address

Frame: Number = 221100, Captured Frame Length = 85, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[DD-DD-DD-DD-DD-DD],SourceAddress:[SS-SS-SS-SS-SS-SS]
+ Ipv4: Src = SS.SS.SS.SS, Dest = DC.DC.DC.DC, Next Protocol = UDP, Packet ID = 0, Total IP Length = 71
- Udp: SrcPort = 60665, DstPort = DNS(53), Length = 51
    SrcPort: 60665
    DstPort: DNS(53)
    TotalLength: 51 (0x33)
    Checksum: 23415 (0x5B77)
    UDPPayload: SourcePort = 60665, DestinationPort = 53
- Dns: QueryId = 0x7DFF, QUERY (Standard query), Query  for SIP.SIP.SIP.SIP of type PTR on class Internet
    QueryIdentifier: 32255 (0x7DFF)
  + Flags:  Query, Opcode - QUERY (Standard query), RD, Rcode - Success
    QuestionCount: 1 (0x1)
    AnswerCount: 0 (0x0)
    NameServerCount: 0 (0x0)
    AdditionalCount: 0 (0x0)
  - QRecord: of type PTR on class Internet
     QuestionType: PTR, Domain name pointer, 12(0xc)
     QuestionClass: Internet, 1(0x1)

Here's the packet sent out by System Process looking for the Suspicious IP address

  Frame: Number = 221101, Captured Frame Length = 92, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[DD-DD-DD-DD-DD-DD],SourceAddress:[SS-SS-SS-SS-SS-SS]
+ Ipv4: Src = DC.DC.DC.DC, Dest = SIP.SIP.SIP.SIP, Next Protocol = UDP, Packet ID = 9293, Total IP Length = 78
- Udp: SrcPort = NETBIOS Name Service(137), DstPort = NETBIOS Name Service(137), Length = 58
    SrcPort: NETBIOS Name Service(137)
    DstPort: NETBIOS Name Service(137)
    TotalLength: 58 (0x3A)
    Checksum: 47118 (0xB80E)
    UDPPayload: SourcePort = 137, DestinationPort = 137
+ Nbtns: Query Request for *<00><00><00><00><00><00><00><00><00><00><00><00><00><00><00> <0x00> Workstation Service

So, at last we were able to track the machine that was forcing our DCs to search for these subnets. The next question was why is this machine doing it? We found out its one of the IP address manager which also scans subnets. Hmm.. that's was interesting so why was this machine looking for external subnet.. apparently the subnet used to be part of our network and was sold off later...Oooppss..

A bit of clean up of the tool resolved the mystery.. Thank God!!