Tool for Scripters/DB devs

[Aokromes]

I think we need a tool to allow scripters and db devs to research sniffs in a better and easier way.

The tool needs to be multiplatform and license must be AGPL.

The idea is you load the parsed sniff into the tool, (it needs to open huge txt files, over 1 GB on size) then:

- 1st dropdown will allow the dev to select what he wants to research (for example: creature, gameobject, vehicle, npc_text, poi, waypoints...)

- 2nd dropdown will list the IDs of the creatures/gameobjects/vehicles you have selected (allows multiple selections).

- 3rd dropdown will list the GUIDs of the creatures/gameobjects of the ID(s) you have selected (allows multiple selections).

When you select the GUID(s) under the dropdowns, you will get the coords for a creature/gameobject or creature using the npc_text or the quest of a poi.

And under that, on the left, you will have the timing(s) where sniffer found the selected object.
When selecting it on the right, we will see the actions performed by the object at that time.

In case of waypoints allow to load maps and draw the route of the creature.

[Shin]

Looks like a good idea, maybe providing a documentation regarding the format of the files that need to be parsed can help a lot whoever is going to implement this

[Malcrom]

The waypoint creater I made does this for waypoints. It's hard to write a program to do everything.

[Carbenium]

@Shin: It is probably advantageous to implement a new machine readable output in WPP since the produced txt files aren't really consitent. That should be the first development step.

[Polaretto]

Quite interesting. Any idea on the language that should be used? 

[Krudor]

C#

[DorekoNeko]

Why C#? I mean. You are asking for someone to do it. Can't it be Java? Or maybe the developer's choice? What's more multiplatform?

 

As for the multiple-selection dropdowns... don't think that's posible without making your own component. I don't really think that part is actually needed. Anyways. I could do this, but I am new to the project and stuff. Any help on how to read the sniffed data would be much appreciated.

 

Thanks.

Edited September 7, 2016 by DorekoNeko

[Nay]

13 hours ago, DorekoNeko said:

Any help on how to read the sniffed data would be much appreciated.

https://github.com/TrinityCore/WowPacketParser (surprise! it's written in C#)

[DorekoNeko]

1 hour ago, Nay said:

https://github.com/TrinityCore/WowPacketParser (surprise! it's written in C#)

If I understood it right, this tool @Aokromes is asking is a separate tool. The only thing in common between the two tools is that one has to process the processed txt file from the other. So, as I said, why the second tool has to be writen in C#? Why can't I do it in e.g. Java? Just want to know. I can do it in C# but I'm just curious.

Edited September 7, 2016 by DorekoNeko

[Nay]

It can be done in any language. It just needs to work and be useful

[jackpoz]

On 6/16/2016 at 4:40 PM, Aokromes said:

The idea is you load the parsed sniff into the tool, (it needs to open huge txt files, over 1 GB on size)

maybe the SQL output of WPP can help this part since it comes down to MySQL performance. After all this tool does not much more than a MySQL SELECT, it just needs data being stored in a more query-friendly way (waypoints drawing is a separated topic than sniff filtering).

[tot78]

For quite some time now, I have been working on a tool called SniffExplorer that does exactly you were looking for: https://github.com/chaodhib/SniffExplorer

It's written in Java and it is very flexible. The downside is that I didn't wrote any GUI so the only way to use the app is by code. On the other hand, the app allows so much flexibility. Some of that flexibility would be hard to expose through any GUI.

A good starting point is here: https://github.com/chaodhib/SniffExplorer/blob/master/src/main/java/com/trinitycore/sniffexplorer/launcher/SniffExplorer.java

Also, here is a few more examples of use: https://github.com/chaodhib/SniffExplorer/tree/master/src/main/java/com/trinitycore/sniffexplorer/launcher

Contributions are welcome. The documentation is pretty much inexistent however I would be happy to answer to questions.

PS: the main target of the app is wotlk sniffs (3.3.5 12340)

-- chaodhib

Edited March 4, 2017 by tot78

[Nay]

Oh noes, parsing arbitrary text 

That is going to break at any minor change to current WPP output format...

[tot78]

Yes, it's parsing the text output of WPP. Sadly, I didn't have any alternative when I started to wrote the app. And as far as I know, WPP still does not support the serialization of structured data. By focusing on wotlk sniffs, I didn't have too much issues regarding changes of the output format. Obviously though, this is clearly a big weakness of the app. And the only way solution is to extend the capabilities of WPP.

Nonetheless, my app has provided with an invaluable source of information. The ability to filter packets by expressing the set of conditions that determined if a packet is relevant to me or not and then getting the result in matter of seconds (as opposed to manual search that can take hours/days) has been incredibly fruitful to me. It also allowed me to test certain assertions (1) quite easily: I just wrote my condition and I let the application go through the entire database of wotlk sniffs that I have at my disposal. Then I let the app do its work while I do something else and I had the result after an hour. Doing the same thing by hand would have taken me days..

 

(1) for example that some packets such as MSG_MOVE_ROOT, MSG_MOVE_WATER_WALK,  MSG_MOVE_HOVER, etc.. are only packets transferred from Server To Client cf https://github.com/chaodhib/TrinityCore/blob/c792f35f0e6492d933c3aabc2cd20ebc42c38aad/src/server/game/Movement/MovementPacketSender.h#L32

[Nay]

I didn't mean to say that it didn't provide value, au contraire, tools like this are absolutely needed.

However, imo, relying on the fragile text output format of WPP is insane; it will change and it does not matter the client version.

The time spent implementing the text parsers could be used to change WPP so that it can output in a better format.

@Polaretto is working on changing this so let's help him

[Polaretto]

We briefly discussed about this on github (link for reference). I'm trying to refactor the output part. Btw I'm assuming that the txt output doesn't have to be *exactly* the same as before.. Am I right? Or changing it would break something?

[Nay]

The "new" text output won't have to be the same and it will break stuff (but that's on the people that assumed that non-formatted text is safe to be consumed).

[Krudor]

Can't you just port the WowPacketParser data model to separate project, and use protobuf to serialize and then deserialize the data in whatever program needs to use it.

[tot78]

If I'm not mistaken, this is what we want to achieve:

step 1) WPP should instantiate and fill one object per packet. The class of the object should depend on the opcode of the packet. This class should model the packet and each class's instance should be able to store every information that could potentially be included in that type of packet.

step 2) serialize each "Packet object" into text format (json, xml, ...).

 

If JSON is chosen, the step 2 is fairly simple in Java once the right library is plugged into the app. I assume it's the same for C#. So step 1 would be most of the work. (https://github.com/chaodhib/SniffExplorer/tree/master/src/main/java/com/trinitycore/sniffexplorer/message/smsg could be used as a starting point).

 

@KrudorI don't think ProtoBuff would be suited for the job regarding the serialization. If I'm not mistaken, with ProtoBuff, one Packet would have to be stored in a seperate file. What would be more suitable for the user would be to serialize every packets contained in the sniff in one single file (otherwise, there would be hundreds of thousands of files). 

 

@Nay Indeed.

Edited March 5, 2017 by tot78

[Nay]

11 hours ago, Krudor said:

Can't you just port the WowPacketParser data model to separate project, and use protobuf to serialize and then deserialize the data in whatever program needs to use it.

WPP does not have a "data model" for packets atm.

 

1 hour ago, tot78 said:

step 1) WPP should instantiate and fill one object per packet. The class of the object should depend on the opcode of the packet. This class should model the packet and each class's instance should be able to store every information that could potentially be included in that type of packet.

 

That's more or less what needs to be done... except that the same packet can have different representations across client versions.

[codeman3518]

On 6/16/2016 at 10:40 AM, Aokromes said:

 

I think we need a tool to allow scripters and db devs to research sniffs in a better and easier way.

The tool needs to be multiplatform and license must be AGPL.

The idea is you load the parsed sniff into the tool, (it needs to open huge txt files, over 1 GB on size) then:

- 1st dropdown will allow the dev to select what he wants to research (for example: creature, gameobject, vehicle, npc_text, poi, waypoints...)

- 2nd dropdown will list the IDs of the creatures/gameobjects/vehicles you have selected (allows multiple selections).

- 3rd dropdown will list the GUIDs of the creatures/gameobjects of the ID(s) you have selected (allows multiple selections).

When you select the GUID(s) under the dropdowns, you will get the coords for a creature/gameobject or creature using the npc_text or the quest of a poi.

And under that, on the left, you will have the timing(s) where sniffer found the selected object.
When selecting it on the right, we will see the actions performed by the object at that time.

In case of waypoints allow to load maps and draw the route of the creature.

 

Aokromes this is really a great idea I could see taking this on and I think this is a project I would be willing to take on as long as people working with this don't have any problems with this being built with Livecode it would be built for windows, mac, and linux it would stay open source of course. I would need a little more information to get this started though, but I like it.

[CDawg]

On 5/4/2017 at 1:13 PM, codeman8214 said:

Aokromes this is really a great idea I could see taking this on and I think this is a project I would be willing to take on as long as people working with this don't have any problems with this being built with Livecode it would be built for windows, mac, and linux it would stay open source of course. I would need a little more information to get this started though, but I like it.

Do eeet!! We need something to make sniffing easier.

When I get time, Iv'e been piecing together the player map. I usually get about 10-15mins a day, due to real life.

[codeman3518]

5 hours ago, CDawg said:

Do eeet!! We need something to make sniffing easier.

When I get time, Iv'e been piecing together the player map. I usually get about 10-15mins a day, due to real life.

That is what worries me about starting projects is the time having to do it because of real life.

[Polaretto]

Hi everyone, 

in the attempt to reach this milestone, I wrote a technical description of the situation up to now on WPP's issue tracker - GitHub.

[Rat]

you know that wpp has an inner storage for everything which is going to be dumped to txt..
so why not simply make a winform gui in wpp? it should not autoclose after parsing is done, instead open up the gui where you could do searching for the already parsed/stored stuff. dont have to reinvent the wheel again, (there is no sniffer atm so you have time  )